Reproduce “OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyue Cai, Chaoyou Fu, Yi-Fan Zhang, Ran He, Caifeng Shan
Researchers developed OmniVideo-100K, an automated pipeline creating a large-scale dataset for audio-visual reasoning, featuring structured scripts and clue-guided QA. Fine-tuning multimodal language models on this dataset led to substantial performance gains on audio-visual benchmarks, enhancing cross-modal and long-term temporal reasoning capabilities by up to 20.59%.
Key insights
- Structured video scripts, guided by a main entity list, effectively reconstruct audio-visual associations and ensure referential consistency across video segments.
- Clue-guided QA generation significantly increases the complexity and temporal span of questions, requiring more profound cross-modal reasoning compared to direct generation.
- The dataset samples necessitate strong cross-modal synergy, as MLLM performance degrades notably when only a single modality is available for inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.