Reproduce “LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ce Zhang, Jinxi He, Katia Sycara, Yaqi Xie

LENS is a training-free framework that dynamically allocates a limited frame budget between spatial zoom-in for fine-grained details and temporal zoom-out for broader context, enhancing multi-modal large language models' (MLLMs) ability to understand long-form videos. It consistently improved video question-answering accuracy by 2.0% to 5.5% on Video-MME, LongVideoBench, and MLVU benchmarks using an 8-frame budget across various MLLM backbones.

Key insights

  • Dynamic, multi-granular keyframe sampling, adapting between fine-grained spatial focus and broader temporal context, is critical for effective long-form video understanding in MLLMs.
  • Leveraging cross-modal attention for query-guided spatial prompting and graph-based reasoning for temporal context allows for a more comprehensive and query-aligned video representation.
  • An adaptive, MLLM-based controller can effectively allocate frame budgets between spatial and temporal emphasis, ensuring efficient resource utilization for diverse queries.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.