Reproduce “Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuchen Cai, Ding Cao, Liang Lin, Chunxi Luo, Xin Xu, Kai Yang, Weijie Liu, Saiyong Yang and 4 more
EffOPD accelerates on-policy distillation by extending the student’s recent parameter movement and retaining candidate extensions that improve a small validation sample. It uses teacher feedback on student-generated answers. On math reasoning, it typically converges in about 10 steps versus 30–40 for vanilla OPD, with faster training in tested settings, not universally.
Key insights
- OPD exhibits 'Functional Redundancy Avoidance', efficiently concentrating parameter updates on modules critical for reasoning (e.g., middle-layer MLPs) while minimizing changes in functionally peripheral regions (e.g., embedding layers).
- OPD demonstrates 'Early Low-Rank Lock-in', where its parameter updates form stronger low-rank structures, and its dominant update directions stabilize earlier and maintain better alignment with the final solution compared to RL.
- Early OPD checkpoints contain high-quality update directions, with performance primarily limited by underdeveloped magnitudes, suggesting that the 'what to update' is learned quickly, while 'how much to update' develops over time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.