Reproduce “Toward Automated Robustness Evaluation of Mathematical Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yutao Hou, Zeguan Xiao, Fei Yu, Yihan Jiang, Ma Shuguang, Zhaoqian Dai, Hailiang Huang, Yun Chen and 1 more
Researchers developed an automated framework, MASTER, to dynamically evaluate Large Language Model robustness in mathematical reasoning by generating adversarial problem variants tailored to induce failures. The framework demonstrated average accuracy drops of 48.04% on GSM8K and 31.35% on MATH-500 across various models, and these generated variants successfully improved model robustness when used for fine-tuning.
Key insights
- LLMs with lower initial accuracy generally exhibit larger test failure rates, though even high-performing models show considerable robustness vulnerabilities under adversarial testing.
- The effectiveness of the MASTER framework scales positively with the capabilities of the Rewriter LLM and successfully identifies robustness issues in stronger target models.
- Adversarial variants generated by MASTER can be used for supervised fine-tuning and direct preference optimization, leading to demonstrable improvements in the robustness of LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.