Reproduce “MMATH: A Multilingual Benchmark for Mathematical Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenyang Luo, Wayne Xin Zhao, Jing Sha, Shijin Wang, Ji-Rong Wen
This research introduces MMATH, a new benchmark for multilingual complex mathematical reasoning across 10 diverse languages, and investigates large language models' performance. It reveals a widespread 'off-target problem' where models often generate responses in unintended languages, but demonstrates that fine-tuning with English reasoning traces while ensuring native language questions and answers (EN-Think strategy) can achieve high accuracy (66.72%) and language consistency (97.61%).
Key insights
- LLMs, particularly dedicated reasoning models, show performance disparities across languages, performing better on high-resource languages but suffering from a prevalent 'off-target problem' in low-resource settings.
- Reasoning in an 'off-target' language (most often English) can surprisingly lead to comparable or superior accuracy for low-resource language problems, suggesting English often serves as an internal reasoning backbone.
- While prompt-based interventions can increase language consistency, they may negatively impact reasoning accuracy, indicating a potential trade-off between explicit language control and natural model reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.