Reproduce “VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiashuo Yu, Yue Wu, Meng Chu, Zhifei Ren, Zizheng Huang, Pei Chu, Ruijie Zhang, Yinan He and 9 more
VRBench introduces the first benchmark for evaluating multi-step reasoning in long narrative videos, featuring a multi-phase evaluation pipeline and human-curated, temporally-grounded annotations across 8 non-English/Chinese languages. This benchmark uncovered "reasoning fragility" in large language and vision-language models, where models often achieve correct outcomes with flawed underlying reasoning, and identified specific weaknesses in tasks requiring fine-grained visual perception and forward-looking inference.
Key insights
- Large models exhibit "reasoning fragility," often achieving high outcome accuracy but with significantly lower quality in their underlying, step-by-step reasoning processes.
- Challenges persist in tasks requiring fine-grained visual perception (e.g., counting specific elements) and forward-looking inference (e.g., event prediction, hypothetical reasoning).
- Long-context support, high frame input rates, and System-2 thinking strategies (e.g., deeper reasoning chains) are crucial for enhancing multi-step reasoning in long narrative videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.