Reproduce “Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinhui Yi, Syed Talal Wasim, Yanan Luo, Muzammal Naseer, Juergen Gall
Researchers from the University of Bonn and Lamarr Institute developed Video-Panda, an encoder-free video-language model that achieves competitive performance on open-ended video question answering while using 6.5x fewer visual parameters and operating 4x faster than prior encoder-based models.
Key insights
- Specialized spatio-temporal modeling within an encoder-free framework is crucial for effective video understanding, outperforming direct extensions of image-based encoder-free models.
- The combination of learnable attention-based spatial downsampling and explicit maintenance of 2D spatial structure via 'row tokens' significantly contributes to efficient performance.
- Distillation using negative cosine similarity proves effective for aligning visual tokens with a teacher model's embedding space, offering robust feature space alignment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.