Reproduce “Frame-Voyager: Learning to Query Frames for Video Large Language Models” on OpenResearch

About this paper

Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun and 4 more

FRAME-VOYAGER introduces a learned, intelligent frame selection mechanism that queries informative combinations of frames for Video Large Language Models, addressing the challenge of LLM token limits and sub-optimal input. The system achieves this by developing a human-free data collection pipeline and consistently improves performance across various video question answering benchmarks with fewer, more relevant frames.

Key insights

• Learning to select query-aware *combinations* of frames is crucial for capturing temporal relationships and global video context, outperforming methods that select individual frames or rely on uniform sampling.
• A human-free data generation pipeline, leveraging pre-trained Video-LLM prediction loss as a ranking signal, effectively creates supervision for complex combinatorial frame selection without manual annotation.
• Intelligently selected, fewer frames can yield better video understanding performance than a larger, uniformly sampled set, demonstrating a 'quality over quantity' approach for efficient Video-LLM input.

Summary by alphaXiv.