Reproduce “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, Gao Huang, Gao Huang
The paper tests whether reinforcement learning with verifiable rewards (RLVR) expands coverage or mainly makes existing solutions easier to sample. Across base and zero-RL models, it finds RL improves pass@1 but often loses at larger k; on Minerva, the Qwen 2.5 32B base exceeds its RL counterpart by about 9 points at pass@128. This result is limited to these models and settings.
Key insights
- Current RLVR primarily improves the *sampling efficiency* of existing reasoning patterns within an LLM, leading to higher `pass@1` accuracy.
- RLVR often *narrows* the overall reasoning coverage of an LLMs, reducing its potential to solve problems when given multiple attempts (`pass@k` for larger `k`).
- Distillation, unlike current RLVR, can genuinely expand a model's reasoning boundary by transferring and introducing new reasoning patterns from a stronger teacher model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.