Reproduce “MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jusheng Zhang, Kaitong Cai, Xiaoyang Guo, Sidi Liu, Qinhan Lv, Ruiqi Chen, Jing Yang, Yijia Fan and 5 more
Researchers from Sun Yat-sen University and industry partners introduced MM-CoT, a diagnostic benchmark designed to evaluate visual grounding and logical coherence in Chain-of-Thought reasoning for multimodal models. The benchmark reveals that state-of-the-art models like GPT-5 achieve only 19.47% accuracy on verification tasks, highlighting a substantial gap between their generative fluency and genuine reasoning fidelity.
Key insights
- Despite their generative fluency, current state-of-the-art multimodal models demonstrate significant deficiencies in verifying true visual grounding and logical coherence, falling far behind human performance.
- MM-CoT successfully differentiates model capabilities, exposing specific challenges in multi-step reasoning for images and cross-frame state tracking for videos.
- The benchmark tasks genuinely require multimodal perception, as removing visual signals dramatically reduces model accuracy, preventing reliance on purely linguistic patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.