Reproduce “MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark” on OpenResearch

About this paper

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu and 5 more

Researchers from Carnegie Mellon University and collaborators developed MMMU-Pro, a more robust benchmark for evaluating multimodal large language models by mitigating shortcut learning and introducing a novel vision-only input setting. State-of-the-art MLLMs exhibited significant performance drops, ranging from 16.8% to 26.9%, on MMMU-Pro compared to previous benchmarks, with reasoning errors for GPT-4o increasing to 46%.

Key insights

• MMMU-Pro effectively exposes the limitations of current MLLMs, causing significant performance drops and highlighting weaknesses in genuine multimodal understanding, especially with integrated vision-text processing.
• Increased candidate options and vision-only inputs significantly challenge models, indicating that high performance on other benchmarks may reflect shortcut learning rather than deep reasoning.
• While Chain of Thought (CoT) prompting can improve performance, its effectiveness varies, and explicit OCR prompts have minimal impact, suggesting that core reasoning and integration are greater challenges than mere text extraction.

Summary by alphaXiv.