Reproduce “A Survey of Reinforcement Learning for Large Reasoning Models” on OpenResearch

About this paper

Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian and 31 more

The authors survey RL for large reasoning models, focusing on training reasoning rather than aligning behavior. They organize the field around reward design, policy optimization, and sampling, with verifiable rewards supplying feedback for checkable tasks. DeepSeek-R1-Zero’s AIME 2024 pass@1 rose from 15.6% to 77.9% during training; this benchmark trajectory is not proof of broad generalization or new abilities.

Key insights

• RL with verifiable rewards (RLVR) provides a scalable path to improve complex reasoning in LLMs, especially for tasks like math and coding, by leveraging automated feedback.
• Effective RL for LRMs relies on a careful combination of dense/unsupervised rewards, efficient off-policy optimization, and dynamic/structured sampling strategies.
• The field is shifting from static datasets to dynamic environments and robust open-source infrastructure, which are crucial for scaling RL training for reasoning tasks.

Summary by alphaXiv.