Reproduce “Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yudi Shi, Shangzhe Di, Qirui Chen, Qinian Wang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Weidi Xie
Weaver, an end-to-end trainable agentic system from Shanghai Jiao Tong University and Xiaohongshu Inc., enhances video reasoning by dynamically acquiring visual evidence through a diverse library of perception tools. It achieves performance gains of up to 12% on long video benchmarks, moving beyond static, text-centric reasoning to a perception-aware approach.
Key insights
- Dynamic, perception-in-the-loop reasoning, which allows models to actively query and integrate visual evidence throughout the reasoning process, is crucial for overcoming limitations like hallucination and static perception in video understanding.
- End-to-end trainability via a combined SFT and RL approach enables an agentic system to learn optimal strategies for tool invocation and composition, leading to superior performance compared to training-free or purely text-centric methods.
- A comprehensive library of specialized visual perception tools, flexibly invoked and combined, empowers the reasoning model with fine-grained spatio-temporal awareness essential for complex and long-form video analysis.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.