Reproduce “Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu and 6 more
SOPD trains students on separate, multi-token teacher targets generated from prefixes in their completed on-policy trajectories. Targets guide the student at states it actually visited but do not form a globally repaired path. On four math benchmarks, SOPD reaches 57.7% average accuracy versus OPD’s 47.7%. In ALFWorld it improves Seen and Unseen success over OPD, but ties OPD on Hard success.
Key insights
- Combining on-policy learning from student-visited states with coherent, segment-level supervision leads to more robust and effective knowledge distillation than either fragmented token-level or off-policy sequence-level methods.
- The ability to provide meaningful multi-token corrections at 'natural step' granularity without interfering with the student's subsequent actions or environment interactions is critical for improving learning.
- SOPD's reliance on only teacher-generated text, rather than logits, makes it compatible with black-box teacher models, broadening the practical applicability of on-policy distillation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.