Reproduce “On the optimization dynamics of RLVR: Gradient gap and step size thresholds” on OpenResearch

About this paper

Joe Suk, Yaqi Duan

Researchers at New York University introduce a theoretical framework for Reinforcement Learning with Verifiable Rewards (RLVR) in large language models, defining a "Gradient Gap" to characterize optimization dynamics and deriving step-size thresholds that govern convergence and divergence. This work explains the efficacy of heuristics like length normalization and validates its findings through simulations and LLM fine-tuning experiments.

Key insights

• The 'Gradient Gap' (difference in expected log-probability gradients between positive and negative response spaces) is directly proportional to the policy gradient, serving as a robust indicator of the true direction for policy improvement.
• A specific step-size threshold exists, beyond which training can lead to catastrophic performance decrease, even with a positive Gradient Gap, emphasizing the critical role of learning rate calibration.
• The effective learning rate for RLVR is theorized to be inversely proportional to response length and sensitive to the current success rate, providing a theoretical basis for the effectiveness of length normalization in algorithms like GRPO.

Summary by alphaXiv.