Reproduce “EvolMathEval: Towards Evolvable Benchmarks for Mathematical Reasoning via Evolutionary Testing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengbo Wang, Mingwei Liu, Zike Li, Anji Li, Yanlin Wang, Xin Peng, Zibin Zheng
The paper introduces EvolMathEval, an automated framework for generating and evolving mathematical reasoning benchmarks for Large Language Models. This system leverages evolutionary testing to produce novel, diverse, and controllable high-difficulty problems, revealing that LLMs frequently prioritize "shortcut" solutions over rigorous reasoning, even leading to performance collapse on evolved tasks.
Key insights
- The data-driven fitness function, incorporating metrics like noise ratio and lexical entropy, reliably correlates with problem difficulty and effectively guides the generation of more challenging tasks.
- Genetic operators, particularly the crossover operator that links two problems sequentially, significantly increase problem difficulty by demanding multi-step reasoning and introducing comprehension pressure.
- LLMs exhibit a prevalent "Pseudo Aha Moment" where they prioritize simplistic, logically flawed "shortcuts" over rigorous, multi-step reasoning, indicating a fundamental limitation in their current capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.