Reproduce “RAGEN-2: Reasoning Collapse in Agentic RL” on OpenResearch

About this paper

Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li and 8 more

The authors study template collapse in multi-turn agent reinforcement learning, where reasoning remains varied but becomes less tied to its input. They measure input dependence with mutual-information proxies and prioritize prompts by within-prompt reward variance. On Sokoban with Qwen2.5-3B, prompt-level filtering scores 23.6%, versus 12.9% without filtering and 16.8% with trajectory-level filtering; its advantage narrows in highly stochastic settings.

Key insights

• Template collapse is a consistent failure mode in agentic RL, characterized by a decline in input dependence (MI) while within-input diversity (entropy) remains elevated, confirming that entropy is an insufficient diagnostic.
• The Signal-to-Noise Ratio (SNR) of policy gradient updates is critical; low reward variance leads to weak task gradients that are overwhelmed by input-agnostic regularization, causing reasoning to become templated.
• Mutual information-based metrics are superior to entropy for monitoring reasoning quality during training, showing a strong positive correlation with task performance and effectively predicting template collapse.

Summary by alphaXiv.