Reproduce “RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training” on OpenResearch

About this paper

Rachit Bansal, Clara Mohri, Tian Qin, David Alvarez-Melis, Sham Kakade

The authors test whether RL can improve checkpoints before SFT. Direct RL uses model-generated math attempts and verifiable answer rewards, not reasoning traces. On GSM8K, a 1B-parameter model’s pass@1 rises from about 2% to about 18% after 4B pre-training tokens, though some early runs fail. In a same-checkpoint comparison, direct RL raises pass@32, while RL after SFT slightly lowers it.

Key insights

• RL can be effective surprisingly early in pre-training, with direct RL matching the performance of the standard SFT→RL pipeline on GSM8K by 10 billion pre-training tokens.
• The 'sharpening effect' (improved pass@1 but decreased pass@k) is an artifact of SFT, while direct RL can expand the output distribution (improving both pass@1 and pass@32) and preserves general model capabilities.
• Targeted pre-training data composition significantly boosts RL efficacy on challenging tasks more than increasing model scale alone, allowing RL to recover qualitative behaviors seen on easier tasks.

Summary by alphaXiv.