Reproduce “Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Runa Yoshida, Kosuke Nishida, Kyosuke Nishida
The authors develop a checklist prompt for mathematical reasoning, based on classifying failures in 583 MATH-test questions answered incorrectly by Llama-3.3-70B-Instruct. It asks models to check conditions, assumptions, theorem prerequisites, and reasoning steps. On 4,300 MATH problems, it reduced incorrect answers versus generic and coarse prompts across three tested models; evidence is limited to MATH and three open-weight models.
Key insights
- The most common failures involved applying inappropriate theorems, ignoring or misinterpreting problem conditions, invalid logical implications, and arithmetic or algebraic manipulation errors; conceptual knowledge mistakes were comparatively less frequent.
- Models often appeared to know the relevant mathematical method but failed to verify its assumptions or maintain constraints throughout the solution.
- Fine-grained, failure-specific checks were more effective than simply telling a model to pay closer attention, suggesting that process diagnostics can directly inform lightweight inference-time interventions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.