Reproduce “Distilling Policy Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, Max Jaderberg
The authors study policy distillation and propose expected entropy regularisation, combining one-step teacher matching with a trajectory-level teacher-likelihood signal. In synthetic grid-world Markov decision processes, student-driven training reached teacher performance in three times fewer optimization steps than teacher-driven training, while the estimator continued learning as irrelevant actions were added. This does not establish performance in deep, real-world reinforcement learning.
Key insights
- Common student-driven distillation updates do not generally form a valid gradient vector field, which can lead to oscillations and non-convergence when combined with environment rewards.
- Student-driven trajectory sampling consistently leads to faster learning and higher student returns compared to teacher-driven or uniform sampling.
- The teacher's value function can be effectively leveraged through intrinsic reward shaping to ensure convergence to optimal policies, even when the teacher is highly suboptimal or adversarial.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.