Reproduce “Progressive distillation induces an implicit curriculum” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, Surbhi Goel
A study investigates why progressive distillation, which leverages intermediate teacher checkpoints, often outperforms one-shot distillation, demonstrating it induces an implicit curriculum that guides student learning through increasingly complex features. For sparse parity tasks, this approach reduces sample complexity from "Ohm(d^k)" to "O(k log d)", and empirically shows consistent performance gains across synthetic and natural language tasks.
Key insights
- Neural networks learn features in distinct phases, progressing from simple to more complex representations during their training.
- Progressive distillation implicitly leverages this natural learning progression, with intermediate teacher checkpoints providing suitable learning targets for student models.
- Intermediate teacher checkpoints, particularly those during phase transitions where new features are learned, often offer optimal supervision for student models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.