Reproduce “Black-Box On-Policy Distillation of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, Furu Wei
GAD lets a student learn from a text-only language-model teacher without access to its probabilities. The student generates answers, while a learned discriminator compares them with teacher responses and scores them; policy-gradient training updates the student as the discriminator adapts. On the held-out LMSYS-Chat test, Qwen2.5-14B-Instruct scores 52.1, versus 50.6 for SeqKD and 51.7 for GPT-5-Chat, under GPT-4o evaluation.
Key insights
- GAD enables mode-seeking behavior in student models, focusing on high-quality outputs, unlike SeqKD which tends to spread probability mass across all possible outputs.
- The co-evolutionary nature of GAD's on-policy discriminator provides stable and robust reward signals, effectively mitigating the reward hacking problem often seen with fixed reward models in RL-based training.
- SeqKD models tend to overfit to local lexical patterns of the teacher, while GAD successfully captures the teacher's broader, global stylistic characteristics, leading to better generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.