Reproduce “M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought” on OpenResearch

About this paper

Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, Wanxiang Che

Researchers from Harbin Institute of Technology, Central South University, and Shanghai AI Laboratory developed M3CoT, a multi-domain, multi-step, multi-modal Chain-of-Thought benchmark. The benchmark reveals a substantial performance gap between state-of-the-art VLLMs and human accuracy, with GPT-4V achieving 62.60% compared to human performance of 91.17%, and its design rigorously enforces visual necessity and complex multi-step reasoning.

Key insights

• Chain-of-Thought (CoT) prompting benefits Vision Large Language Models (VLLMs) only if they have parameters exceeding 10 billion, indicating CoT as an emergent ability tied to model scale.
• Performance of VLLMs on multi-step multi-modal reasoning drastically declines as the number of required reasoning steps increases, suggesting this is a challenging Out-of-Distribution problem for current models.
• Effective In-Context Learning (ICL) for multi-modal tasks relies heavily on high-quality, genuinely interleaved image-text examples, with insufficient quality or interleaving potentially degrading performance.

Summary by alphaXiv.