Reproduce “Reinforcement Learning for Reasoning in Large Language Models with One Training Example” on OpenResearch

About this paper

Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He and 6 more

The authors test whether one distinct math problem, reused during reinforcement learning with a verifiable reward (RLVR), can improve mathematical reasoning. They train Qwen2.5-Math-1.5B with GRPO on a selected problem, rewarding correct answers. MATH500 accuracy rises from 36.0% to 73.6%, and the six-benchmark average from 17.6% to 35.7%; this result concerns selected examples and does not show reduced computation.

Key insights

• LLMs possess significant latent reasoning capabilities that 1-shot RLVR can effectively activate with extremely sparse data, suggesting RLVR primarily acts as a catalyst for existing skills rather than teaching entirely new ones.
• The phenomenon of "post-saturation generalization" was identified, where test performance continues to improve long after training accuracy saturates and even when the model overfits, distinguishing it from traditional "grokking."
• Policy gradient loss is the primary driver of reasoning improvements, while entropy loss plays a critical role in promoting diverse exploration and enhancing sustained generalization across tasks.

Summary by alphaXiv.