Reproduce “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, Yi Dong

NVIDIA researchers developed Prolonged Reinforcement Learning (ProRL) to expand large language models' reasoning capabilities beyond their initial pre-training. This method, featuring KL divergence control and reference policy resetting on diverse tasks, enables significant performance gains, including average pass@1 improvements of up to 54.8% on logic puzzles, and the discovery of novel solution pathways.

Key insights

  • Reinforcement learning, when prolonged and carefully managed, can genuinely expand LLM reasoning capabilities and uncover novel solution pathways, refuting prior skepticism about its limitations.
  • Extended training duration, coupled with diverse task exposure, is crucial for fostering exploration, consolidating new strategies, and achieving significant performance gains.
  • The most substantial reasoning improvements from RL are observed in tasks where the base model initially performs poorly, indicating effective boundary expansion for challenging problems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.