Reproduce “Agentic Keyframe Search for Video Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sunqi Fan, Meng-Hao Guo, Shuojin Yang
Tsinghua University researchers developed Agentic Keyframe Search (AKEYS), an approach that uses Large Language Models as agents to guide the search for relevant video frames to answer questions. This method achieves state-of-the-art accuracy on VideoQA benchmarks while processing only about 15% of total frames, addressing computational and context window limitations of traditional Video-LLMs.
Key insights
- Leveraging LLMs as intelligent agents to guide keyframe search based on specific questions significantly improves efficiency and accuracy in VideoQA.
- Framing keyframe search as a graph search problem on a video segment tree allows for adaptive and question-oriented frame selection.
- Combining heuristic guidance (relevance to question) with movement cost (scene changes) in search strategies like A*-search yields superior performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.