Reproduce “Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Vaggelis Dorovatas, Soroush Seifi, Gunshi Gupta, Rahaf Aljundi

A training-free and model-agnostic method for efficient streaming video-LLMs from researchers at Toyota Motor Europe, the University of Oxford, and Athena RC, named rLiVS, significantly improves video question answering. It intelligently selects visual tokens using LLM attention, maintains temporal context with recurrent memory, and retrieves textual captions for answering, achieving state-of-the-art performance on streaming video QA benchmarks with reduced memory usage and latency.

Key insights

  • LLM's internal attention mechanisms can effectively guide visual token selection, enabling aggressive compression (up to 95% token discard) with minimal impact on performance.
  • A recurrent memory (FIFO queue) of selected visual tokens is crucial for maintaining temporal coherence and improving performance in long video understanding tasks.
  • LLMs demonstrate superior performance in answering questions by reasoning over retrieved textual captions rather than direct visual tokens, aligning with their proficiency in extended textual contexts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.