Reproduce “Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, Sean Hendryx

This paper investigates how large language models learn complex reasoning when trained with Reinforcement Learning with Verifiable Rewards (RLVR), formally decomposing performance gains into 'self-distillation' and 'capability gain'. The authors introduce 'Guide-GRPO,' an adaptive online RL algorithm that strategically injects natural language hints when unguided attempts fail, demonstrating superior generalization on mathematical reasoning benchmarks and a consistent performance lead across model scales up to 72B parameters.

Key insights

  • Reinforcement Learning with Verifiable Rewards (RLVR) performance gains are predominantly driven by 'self-distillation,' where models learn to convert problems solvable with multiple attempts (pass@k) into single-attempt solutions (pass@1).
  • Genuine 'capability gain,' solving problems previously unsolvable even with multiple attempts, contributes a smaller but present portion to RLVR's learning progress across various model scales.
  • Adaptive, in-context guidance applied selectively when all unguided rollouts fail, combined with importance sampling, accelerates the acquisition of new problem-solving abilities without requiring hints at inference time.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.