Reproduce “OPD+: Rethinking the Advantage Design for On-Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanyang Zhao, Haoxian Chen, Han Lin, Genta Indra Winata, David Yao, Wenpin Tang
A study by Columbia University and collaborators re-examines On-Policy Distillation (OPD) for large language models, correcting the design of advantage functions to mitigate biases from stop-gradient applications. The resulting OPD+ framework prevents catastrophic failure for various f-divergences and yields superior or competitive performance on mathematical reasoning and tool-use tasks compared to existing methods.
Key insights
- Applying stop-gradients to the reward term in OPD results in biased policy gradient estimates when the reward implicitly depends on the student's policy parameters.
- The "catastrophic collapse" observed with f-divergences like Forward KL and Jensen-Shannon Divergence in traditional OPD is due to this gradient bias, and they can be "rescued" by the corrected advantage function.
- The corrected advantage function, even for Reverse KL, can provide empirical improvements by acting as an effective variance-reduction baseline during training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.