Reproduce “VCA: Video Curious Agent for Long Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, Chuang Gan
Researchers at the University of Massachusetts, Amherst and MIT propose VCA, a self-exploration agent framework that mimics human curiosity for efficient long video understanding. The system achieves higher accuracy on benchmarks like EgoSchema and LVBench while processing significantly fewer frames compared to direct VLM application and other agent systems, making complex video analysis more practical.
Key insights
- A tree-search exploration strategy enables the agent to iteratively refine its focus from coarse video segments to finer, more relevant ones, mimicking human-like 'zoom-in' behavior.
- An intrinsic reward model, powered by the same VLM, guides the agent's exploration by self-generating relevance scores for video segments to the query, eliminating the need for external feedback or separate training.
- A fixed-size memory buffer with intelligent management discards less relevant frames while retaining their textual descriptions, allowing for efficient long-term reasoning within constant computational resources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.