Reproduce “When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better” on OpenResearch

About this paper

Siyan Zhao, Yonggan Fu, Jindong Jiang, Shih-Yang Liu, Song Bian, Byung-Kwan Lee, Sharath Turuvekere Sreenivas, Wenliang Dai and 3 more

Semi-OPD caches responses from an initial student and teacher scores them once, then recomputes student probabilities and token-level advantages during training. Across 17 tested teacher–student pairs, it outperforms OPD on 14; the other three are close. For one Qwen3 pair, it scores 13.1 percentage points higher at the final shared step. This is not universal: initial trajectory quality and alignment matter.

Key insights

• Teacher–student alignment helps predict which approach works better: lower initial top-64 overlap is associated with larger Semi-OPD gains over on-policy distillation (Pearson r = −0.73).
• The source of cached data matters: teacher-generated responses performed worse than initial-student responses, and truncating Semi-OPD rollouts reduced performance.
• Semi-OPD depends on a capable starting student; it can underperform on-policy distillation when the initial model is weak.

Summary by alphaXiv.