Reproduce “FOCUS: Efficient Keyframe Selection for Long Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu, Kanchan Sarkar, Zhenheng Yang, Yang You
FOCUS selects query-relevant frames from long videos for multimodal language models with a fixed budget. It divides videos into clips, uses BLIP scores and bandit-style search to prioritize promising or uncertain regions, then selects frames. On LongVideoBench videos over 20 minutes, it achieves 63.7% accuracy with 64 frames, versus 51.8% for uniform sampling; proxy scores and independence assumptions limit interpretation.
Key insights
- Adaptive sampling, guided by multi-armed bandit principles, effectively allocates computational resources to explore and exploit potentially informative temporal regions in long videos.
- A two-stage exploration-exploitation approach provides a practical, parallelizable solution for MAB-based keyframe selection, overcoming the inefficiencies of sequential methods.
- Leveraging temporal locality in videos, where adjacent frames tend to have similar relevance, allows for efficient estimation of clip-level relevance without exhaustively scoring every frame.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.