Reproduce “Speculative Self-Distillation enables Efficient Knowledge Internalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shayan Talaei, Agam Bhatia, Arshia Soltani Moakhar, Jonas Hübotter, Amin Saberi, Azalia Mirhoseini
The paper introduces selective self-distillation (SSD), training a model to answer using context-informed teacher guidance. SSD uses the teacher’s next token when its prediction differs from the student’s beyond a threshold, while applying distillation loss at every response position. Across three QA tasks, it uses 38–69% fewer supervised tokens than on-policy training, with accuracy within 1.4 percentage points except on Company Memo.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.