Reproduce “Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai and 1 more
AOPD extends on-policy distillation by reinforcing student-sampled tokens that a teacher favors and matching the teacher’s top-32 next-token distribution when feedback is non-positive. On Qwen3-8B-Base after 1,000 supervised fine-tuning steps, it reaches 43.13% average Pass@1 versus 34.79% for OPD across three math benchmarks. This is an aggregate result, not every metric.
Key insights
- The method assigns different learning mechanisms to different token states: policy gradients exploit favorable student behavior, while forward KL repairs weak, ambiguous, or teacher-disfavored choices.
- Probability-gap-based guidance avoids the heavy-tailed updates associated with log-probability advantages and helps prevent entropy collapse while preserving exploration.
- Localized teacher correction can address an exploration black hole that global suppression methods leave unresolved, especially when the student has weak initialization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.