Reproduce “Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Wang, Yuxiang Nie, Yongjie Ye, Deng GuanYu, Yanjie Wang, Shuai Li, Haiyang Yu, Jinghui Lu and 1 more
Bytedance Inc. researchers developed Dynamic-VLM, a Large Vision-Language Model for video that introduces a dynamic visual token compression architecture and a large-scale synthetic video-text dataset. This model efficiently processes videos of varying lengths while preserving detail, achieving state-of-the-art performance in video question-answering and demonstrating robust zero-shot generalization to multi-image tasks.
Key insights
- Dynamic visual token compression, particularly using Adaptive Spatial Pooling, effectively balances fine-grained frame detail and extensive temporal coverage, significantly improving video understanding across varied lengths.
- High-quality synthetic video-text data, generated by advanced proprietary models like GPT-4V/4o with carefully engineered prompts, is critical for bridging the data gap in open-source VideoLLM development.
- Training on comprehensive video data, encompassing temporal relationships and context across frames, can instill robust visual and semantic understanding that generalizes strongly to out-of-domain multi-image reasoning tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.