Reproduce “MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro, Tadesse Destaw Belay, Catherine Nana Nyaah Essuman, Ifeoma Okoh, Ganiyat Afolabi, Ayodele Awokoya and 1 more

MGSM-Pro evaluates multilingual math models on controlled versions of grade-school problems, preserving their arithmetic structure while varying digits and, in some versions, names or irrelevant context. The authors test nine languages and recommend five digit-varying versions per problem. Gemini 2.5 Flash averages 85.4% on original data and 71.1% on a digit-and-context variant; results vary by language group.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.