Reproduce “Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs” on OpenResearch

About this paper

Yuanjie Lyu, Chengyu Wang, Jun Huang, Tong Xu

Researchers developed SCoRe (Student-Centered One-step Reinforcement), a distillation framework that enables smaller language models to achieve agentic capabilities comparable to much larger models. A Qwen2.5-7B-Instruct student model trained with SCoRe-RL attained an average score of 52.3 on reasoning benchmarks, rivaling 72B teacher and 671B DeepSeek-V3 models, while significantly improving performance on deep search tasks.

Key insights

• A student-centered, minimal intervention approach generates capability-matched training data that efficiently addresses student deficiencies and significantly reduces cumulative error propagation.
• Shortening RL rollouts by starting from teacher-verified prefixes substantially decreases the variance of policy gradient estimation, leading to more stable and effective reinforcement learning for LLM agents.
• Combining targeted SFT with carefully designed RL, including key-step rewards, allows smaller models to develop genuine problem-solving strategies, even tackling tasks that were initially 'Hard-to-Teach' for the teacher's one-step corrections.

Summary by alphaXiv.