Reproduce “Distillation Scaling Laws” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, Russ Webb
The authors propose scaling laws for choosing whether to train a language-model student directly or distill it from a teacher. The laws relate student validation cross-entropy to student size, distillation tokens, and teacher cross-entropy. Across transformer models on English C4, predictions had at most about 1% relative error. Distillation can lose when teacher training is charged to one student.
Key insights
- The influence of a teacher model's size and training tokens on student performance is entirely summarized by its resulting teacher cross-entropy (L_T).
- The proposed scaling law intrinsically captures and explains the 'capacity gap,' where improving teacher performance beyond a certain point can paradoxically degrade student performance due to a mismatch in learning capacity.
- Pure distillation (λ=1) and a distillation temperature of τ=1 consistently lead to optimal student performance in this experimental setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.