Reproduce “T*: Re-thinking Temporal Search for Long-Form Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles and 4 more
Researchers from Stanford, Northwestern, and Carnegie Mellon Universities introduced T*, a framework that rethinks temporal search by reframing it as a spatial search task to efficiently understand long-form videos. T* substantially improved the QA accuracy of large vision-language models while drastically reducing computational cost and frame usage on natural long-form videos.
Key insights
- Reframing temporal search as a spatial search problem allows leveraging the strengths of image-language models and visual localization for more efficient video understanding.
- An iterative search strategy with adaptive sampling and probability distribution updates effectively 'zooms in' on crucial temporal regions, resembling a quaternary search with logarithmic complexity.
- Temporal F1 and Temporal Precision metrics show the highest correlation with downstream task accuracy, indicating their effectiveness in evaluating the quality of temporal search.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.