Reproduce “Knowledge Distillation for Language Models: A Survey of Training Contexts and Supervision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hoang Tran Vuong, Quoc Phong Dao, Dat Dao, Chi Khanh Pham, Minh-Phuc Truong, Phat Tien Nguyen, Quyen Tran, Minh Le and 6 more
The survey classifies knowledge distillation by training-context origin and, for student-generated contexts, by whether the generating checkpoint is current or earlier. It separates context provenance from teacher supervision and applies the framework across generation, embeddings, retrieval, and agents. A reported Lightning OPD 8B setup uses 30 rather than 120 GPU-hours, but this does not test the taxonomy experimentally.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.