Reproduce “Knowledge Distillation from A Stronger Teacher” on OpenResearch

About this paper

Tao Huang, Shan You, Fei Wang, Chen Qian, Chang Xu

A knowledge distillation method, DIST (Distillation from A Stronger Teacher), effectively transfers knowledge from powerful teacher models to smaller student models by preserving inter-class and intra-class prediction relations using Pearson correlation. The approach consistently improves student performance across image classification, object detection, and semantic segmentation, notably outperforming existing KD techniques when teachers employ advanced training strategies.

Key insights

• Strong teacher models trained with advanced strategies can produce highly nuanced predictions that are difficult for smaller student models to precisely replicate, leading to performance degradation with strict matching objectives.
• Preserving the *relative preferences* or *rankings* of predictions (relational knowledge) is more robust and beneficial for knowledge transfer from strong teachers than attempting an exact match of absolute probability values.
• Both inter-class relations (within-instance class preferences) and intra-class relations (across-instance class typicality) provide valuable, complementary semantic knowledge for distillation.

Summary by alphaXiv.