Reproduce “Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao and 1 more
Researchers at the Beijing Academy of Artificial Intelligence (BAAI) developed Video-XL-2, a multimodal large language model that uses task-aware Key-Value sparsification to efficiently understand very long videos. The model processes over 10,000 frames on a single 80GB A100 GPU, achieving state-of-the-art performance on various benchmarks while significantly reducing computational and memory costs.
Key insights
- Bi-level KV decoding effectively balances memory efficiency and detail preservation by dynamically loading dense KVs for relevant video chunks and sparse KVs for less relevant ones, guided by query relevance.
- Chunk-based pre-filling, which attends to historical timestamp tokens across chunks and uses a sliding window for local context, scales linearly with video length, overcoming quadratic attention costs.
- Integrating explicit timestamp tokens and implicit soft time embeddings within the DTS module significantly enhances the model's temporal awareness for long video understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.