Reproduce “On-Policy Distillation with Negative-Policy Rollouts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaehui Hwang, Dongyoon Han, Sangdoo Yun, Byeongho Heo
NP-OPD trains a student on its own text and weaker-policy rollouts, applying the same token-level teacher feedback to both. The rollouts add a move-away reference without a rejection loss. In non-thinking Qwen3-1.7B, Math Avg rose from 48.3 to 56.3 using Qwen3-0.6B rollouts; this single result does not establish gains on every task or setting.
Key insights
- The method does not assume every negative-policy token is wrong: teacher-favored tokens can still receive positive reinforcement, while the overall rollout distribution supplies a negative reference.
- Analyses show stronger suppression of tokens preferred by the negative policy over the teacher, consistent with NP-OPD shifting predictions away from the weaker policy.
- The choice of negative policy matters: simply degrading student rollouts through decoding or prompting yielded smaller average gains than using a separate lower-capability model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.