Reproduce “LongVILA: Scaling Long-Context Visual Language Models for Long Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang and 10 more

LongVILA, from NVIDIA and academic partners, is a visual language model that scales to understand extremely long videos by integrating a multi-stage training pipeline with a new distributed system. It processes thousands of video frames, achieving 99.8% accuracy on 6,000-frame videos (over 1 million tokens), and its Multi-Modal Sequence Parallelism system provides up to 5.7x faster training throughput compared to existing methods.

Key insights

  • A "full-stack co-design" approach, integrating algorithmic innovations with system-level optimizations, is essential for scaling multi-modal models to long contexts.
  • Extending the large language model's context length with text-only data before multi-modal long-context fine-tuning improves visual language model performance on long videos.
  • Optimizing distributed training for multi-modal data requires a two-stage sharding strategy and a 2D-Attention mechanism to effectively handle modality and networking heterogeneity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.