Reproduce “Clapper: Compact Learning and Video Representation in VLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lingyu Kong, Hongzhi Zhang, Jingyuan Zhang, Jianzhao Huang, Kunze Li, Qi Wang, Fuzheng Zhang
The paper introduces Clapper, an efficient architecture for video understanding in Vision-Language Models, by employing a 'slow-fast' strategy and a specialized TimePerceiver module. This approach achieves over 4x compression in visual tokens while maintaining competitive performance across multiple video understanding benchmarks.
Key insights
- A slow-fast processing strategy effectively balances spatial detail preservation with efficient temporal information extraction in video understanding.
- A specialized TimePerceiver module can efficiently capture temporal dynamics by focusing on changes through cross-attention, reducing token count while maintaining relevant information.
- A two-stage training approach, pre-training the temporal module before end-to-end fine-tuning, proves effective for optimizing video Vision-Language Models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.