Reproduce “Fixing the Teacher-Student Knowledge Discrepancy in Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiangfan Han, Mengya Gao, Yujie Wang, Quanquan Li, Hongsheng Li, Xiaogang Wang
Researchers from CUHK and SenseTime developed Knowledge Consistent Distillation (KCD), a method that addresses the internal feature representation misalignment between teacher and student models by applying a channel-based transformation to teacher features. This approach improved student model performance across image classification and object detection tasks, enhancing existing distillation techniques and leading to more accurate, efficient smaller neural networks.
Key insights
- The internal feature representation discrepancy between teacher and student models, even with similar architectures, is a critical, often overlooked, barrier to effective knowledge distillation.
- A static, student-dependent channel-based transformation (especially using bipartite matching) effectively aligns teacher knowledge, enhancing distillation without complex dynamic updates.
- Consistent student initialization between the transformation calculation phase and the final distillation training is crucial for the KCD method's effectiveness, validating its student-specific nature.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.