Reproduce “ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tanveer Hannan, Md Mohaiminul Islam, Jindong Gu, Thomas Seidl, Gedas Bertasius

ReVisionLLM introduces the first Vision-Language Model capable of accurately performing temporal grounding in hour-long videos. It leverages a recursive, coarse-to-fine processing architecture and a progressive training strategy, achieving new state-of-the-art performance on datasets like MAD and VidChapters-7M.

Key insights

  • A recursive, coarse-to-fine processing strategy, inspired by human cognition, can effectively scale Vision-Language Models to hour-long videos while preserving temporal granularity.
  • Training with contrastive video segments (where the event is absent) significantly improves a VLM's confidence calibration, reducing false positives in long-video temporal grounding.
  • Leveraging the LLM's internal confidence (derived from prediction entropy) provides a more robust and reliable method for ranking temporal event predictions than heuristic similarity scores.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.