Reproduce “Learning to Solve Hard Problems in RL for LLMs by Never Giving Up” on OpenResearch

About this paper

Michael Noukhovitch, Hamish Ivison, Nathan Lambert, Aaron Courville

Never Give Up, or NGU, reallocates reinforcement-learning sampling by retrying prompts whose initial completions are all wrong, while filtering already-solved prompts and using asynchronous training. The authors identify a Matthew Effect: reinforcement learning improves easier problems more than harder ones across math, code, and agentic coding. On Deepscaler, NGU improved hard-subset pass@1 by 4.3 ± 1.2 percentage points versus 2.5 ± 1.1 for fixed-K GRPO, under one matched setup.

Key insights

• The Matthew Effect is driven partly by compute allocation: large fixed groups can keep easy prompts active because a single failed sample creates a training signal, whereas hard prompts may produce no useful signal at all.
• Increasing the fixed group size does not reliably solve hard-problem learning; effective sampling must balance the chance of discovering a correct completion against redundant generation on already-mastered prompts.
• Difficulty is dynamic during RL, so static curricula based on initial model performance can be less effective than adaptive sampling that responds to the model’s current outcomes.

Summary by alphaXiv.