Reproduce “TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chuyi Shang, Amos You, Sanjay Subramanian, Trevor Darrell, Roei Herzig
TraveLER, a modular multi-LMM agent framework from the University of California, Berkeley, introduces an adaptive, iterative reasoning process for video question-answering, enabling models to intelligently navigate videos and refine information. This framework improves zero-shot accuracy on benchmarks such as NExT-QA by +0.5% and EgoSchema (Full) by +3.0%, often while processing significantly fewer frames.
Key insights
- Iterative planning and reflection significantly enhance video understanding by allowing dynamic adaptation and refinement of information collection based on ongoing progress.
- Active, question-driven information extraction at the frame level yields more specific and relevant visual details, which is crucial for answering nuanced causal, temporal, or descriptive queries.
- A modular, multi-agent architecture provides flexibility and extensibility, allowing for easy integration of different underlying LLMs and LMMs, and promoting efficient resource allocation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.