Reproduce “Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Youliang Yuan, Qiuyang Mang, Jingbang Chen, Hong Wan, Xiaoyuan Liu, Junjielong Xu, Jen-tse Huang, Wenxuan Wang and 2 more
A Rubric Reward Model (RRM) is introduced to mitigate "false positives" in Large Language Model mathematical reasoning, which occur when correct answers are reached through flawed logical steps. This process-oriented feedback mechanism improves Verified Pass@N on mathematical benchmarks, boosting AIME2024 performance by 35.9 points and reducing instances of "Miracle Steps" errors by 71% compared to outcome-based training.
Key insights
- "Miracle Steps," where models produce correct intermediate or final answers without valid preceding derivations, are a critical and prevalent false positive, likely linked to answer memorization from pretraining.
- False positives are a systemic issue affecting even leading state-of-the-art LLMs, suggesting that simply scaling model size or training data does not inherently eliminate unfaithful reasoning.
- Process-oriented, rubric-based rewards are more effective at identifying and mitigating specific reasoning flaws than generic process reward models or outcome-only supervision.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.