Reproduce “RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning” on OpenResearch

About this paper

Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, Guorui Zhou

Researchers at Kuaishou Technology's Klear Team developed RLEP, a two-phase reinforcement learning framework that significantly accelerates convergence and improves the final performance of Large Language Models in reasoning tasks. The method achieves this by first collecting high-quality, verified reasoning paths from a converged policy, then replaying these successes uniformly alongside new rollouts during a restarted training phase, demonstrating gains up to 5.2 percentage points on mathematical reasoning benchmarks.

Key insights

• Leveraging high-quality, verified successful trajectories collected from a converged policy provides a stable and efficient learning path for LLM reinforcement learning.
• Uniformly interleaving replayed successes with newly generated on-policy rollouts effectively guides the model away from unproductive exploration and concentrates learning on fruitful reasoning paths.
• Asymmetric clipping and token-mean averaging in the policy optimization help preserve granular learning signals and maintain a balance between exploitation and exploration.

Summary by alphaXiv.