Reproduce “Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It” on OpenResearch

About this paper

Yupu Hao, Zhuoran Jin, Huanxuan Liao, Kang Liu, Jun Zhao

The paper examines structural failure in multi-step tool use during reinforcement learning, where malformed control-token sequences can stop valid tool calls. It tests targeted supervision interleaved with RL, including Process Reflection Supervision (PRS) and Erroneous Trajectory Supervision (ETS). On BFCL-V3, Qwen2.5-1.5B-Instruct with PRS scores 25.75 versus 3.50 vanilla and 0.00 GRPO. On Qwen3-1.7B, ETS scores 23.25 versus PRS 19.5; OOD robustness remains limited.

Key insights

• The catastrophic collapse in pure RL for multi-step tool use is a structural issue caused by the over-amplification and incorrect combination of special control tokens, not a degradation of the model's intrinsic reasoning abilities.
• Interleaved training paradigms, which alternate supervisory signal updates with RL exploration, are more effective at stabilizing training and improving performance than synchronous integration methods.
• Process-level supervisory signals, such as Process Reflection Supervision (PRS), improve generalization by encouraging the LLM to internalize logical scaffolding rather than solely optimizing for output format.

Summary by alphaXiv.