Reproduce “Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, Qian Zhang
The authors propose per-example weighting for knowledge distillation, scaling each teacher-guided loss according to student and teacher one-hot cross-entropies. This retains soft-target guidance while reducing its contribution for selected examples. On ImageNet validation, a ResNet-18 student distilled from a ResNet-34 teacher reaches 72.04% top-1 accuracy, versus 70.67% with standard distillation; this measures accuracy, not speed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.