Reproduce “Self-Distilled Agentic Reinforcement Learning” on OpenResearch

About this paper

Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu and 3 more

SDAR trains long-horizon agents by keeping verifier-driven reinforcement learning as the main objective and adding token-level self-distillation from a privileged-context version of the policy. A detached sigmoid gate gives more weight to tokens the teacher scores above the student and less to those it scores below. On Qwen2.5-3B-Instruct ALFWorld, SDAR reaches 84.4% versus GRPO’s 75.0% overall success rate; evidence spans several benchmarks and includes exceptions.

Key insights

• The privileged teacher's guidance in OPSD is often asymmetric; positive endorsements are valuable, but 'negative rejections' from lower-probability tokens can be misleading due to imperfect skill retrieval or multi-turn drift.
• Dynamically gating token-level distillation based on the teacher-student log-probability gap or student uncertainty allows for adaptive knowledge transfer, prioritizing beneficial signals while suppressing potentially harmful ones.
• Maintaining RL as the primary optimization objective, with OPSD providing a carefully calibrated, gated auxiliary signal, is crucial for stable and effective training of multi-turn LLM agents.

Summary by alphaXiv.