Reproduce “DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anhao Zhao, Haoran Xin, Junlong Tong, Yingqi Fan, Xuan Lu, Ping Nie, Wenjie Li, Xiaoyu Shen
DIAL-OPD selects which student-generated tokens contribute to on-policy distillation loss, using both teacher–student disagreement and the probabilities’ absolute scale. This sampled-token approach avoids full-vocabulary probabilities, while retaining all tokens in context. In tested Qwen3 math experiments, retaining 40% of response positions improved seven-benchmark mean accuracy over full-token Vanilla OPD, with a largest reported gain of 5.25 percentage points; this does not establish lower training time or compute.
Key insights
- In the tested settings, sparse supervision could outperform dense supervision: random retention at several rates beat full-token OPD in an initial experiment.
- Large log-ratio rewards can be misleading when both models assign a token low probability; at 10% retention, low-low tokens made up 20.59% of selected tokens but 49.80% of their absolute reward.
- Moderate probability weighting worked better than either ignoring probability scale or applying it too strongly; the best β varied by student size.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.