Reproduce “BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiheng Xi, Guanyu Li, Yutao Fan, Honglin Guo, Yufang Liu, Xiaoran Fan, Jiaqi Liu, Jingchao Ding and 8 more
Fudan University and collaborators introduce BMMR, a large-scale, bilingual (English and Chinese), multimodal, multi-discipline dataset for evaluating and training Large Multimodal Models (LMMs) and Large Reasoning Models (LRMs) at a college level. The dataset includes 110k instances with detailed reasoning paths and features a process-based verifier, revealing significant headroom for LMM improvement with leading models scoring around 50% on the evaluation set.
Key insights
- Current state-of-the-art LMMs, including leading proprietary models, exhibit substantial headroom for improvement on BMMR's college-level multidisciplinary reasoning tasks.
- Chain-of-Thought (CoT) reasoning consistently improves LMM performance, but reasoning-focused models (LRMs) show discipline biases, excelling in technical fields but lagging in humanities.
- Error analysis reveals that disciplinary knowledge, calculation, derivation, and reasoning errors are the dominant types, indicating models struggle with complex System-2 reasoning and specialized knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.