Reproduce “On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement” on OpenResearch

About this paper

Wenlong Deng, Yushu Li, Boying Gong, Yi Ren, Christos Thrampoulidis, Xiaoxiao Li

The paper examines instability in search-agent training with GRPO, where reward can rise as correct-response likelihood falls before gradients surge and training collapses. It proposes LLDS, which penalizes decreased-likelihood tokens in actions whose likelihood has fallen. On seven QA benchmarks, Qwen2.5-7B-Base scores 0.480 versus 0.350 for Search-R1-GRPO; this comparison uses baseline’s best checkpoint before collapse and covers search-integrated QA.

Key insights

• Lazy Likelihood Displacement (LLD) is the core, previously overlooked, mechanism driving GRPO training collapse in tool-integrated reinforcement learning.
• The LLD 'death spiral' is a self-reinforcing process where declining likelihoods inflates gradients from incorrect trajectories, further accelerating likelihood decay and instability.
• Two factors unique to tool-integrated settings, frequent correct actions within incorrect responses and low-likelihood reasoning after out-of-distribution tool feedback, amplify LLD.

Summary by alphaXiv.