Reproduce “LLaVA-Video: Video Instruction Tuning With Synthetic Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, Chunyuan Li
Researchers developed a pipeline to generate LLaVA-Video-178K, a synthetic dataset featuring 1.3 million instruction samples for video-language models, addressing the scarcity of high-quality video data. The resulting LLaVA-Video family of models demonstrates strong performance, often surpassing existing baselines and competing with commercial models on 11 video understanding benchmarks.
Key insights
- The quality of video instruction-following data, characterized by dynamic content, untrimmed videos, and dense temporal annotations, is more critical for robust video LMM training than sheer data quantity.
- Utilizing a larger number of densely sampled frames (up to 110 frames in experiments) significantly and consistently improves video understanding capabilities across various tasks.
- A recurrent, multi-level description generation pipeline effectively captures temporal coherence and provides rich narratives for long video segments, enabling detailed instruction tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.