Reproduce “Revisiting Knowledge Distillation via Label Smoothing Regularization” on OpenResearch

About this paper

Li Yuan, Francis E.H.Tay, Guilin Li, Tao Wang, Jiashi Feng

This work challenges the conventional understanding of Knowledge Distillation (KD), proposing that its effectiveness largely stems from the regularization effect of soft targets, not solely from inter-category similarity transferred by a strong teacher. Researchers establish a theoretical link between KD and Label Smoothing Regularization (LSR) and introduce Teacher-free Knowledge Distillation (Tf-KD) methods that achieve performance comparable to or exceeding traditional KD and LSR, without requiring a pre-trained superior teacher.

Key insights

• The 'dark knowledge' in Knowledge Distillation is not exclusively inter-category similarity but significantly, or even predominantly, the regularization effect provided by soft targets.
• Knowledge Distillation and Label Smoothing Regularization are fundamentally related, with KD being a form of learned label smoothing and LSR being an ad-hoc distillation.
• Even models with significantly lower accuracy (e.g., poorly-trained teachers or weaker students) can provide beneficial regularization signals in a distillation context.

Summary by alphaXiv.