Reproduce “Improved Knowledge Distillation via Teacher Assistant” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, Hassan Ghasemzadeh
Teacher-assistant knowledge distillation (TAKD) trains an intermediate model from a large teacher, then uses it to train a fixed, smaller student. In tested image-classification setups, student accuracy could rise and then fall as teacher size increased. On CIFAR-10, TAKD reached 73.51% top-1 test accuracy, versus 72.57% for direct distillation. This is evidence from tested setups, not a universal rule.
Key insights
- A significant capacity gap between a large teacher and a small student can lead to performance degradation in direct knowledge distillation.
- Introducing intermediate Teacher Assistant (TA) networks effectively bridges this capacity gap, leading to improved knowledge transfer and a flatter, more generalizable loss landscape for the student.
- The optimal Teacher Assistant size is not necessarily the arithmetic average of teacher and student sizes, but rather aligns with the average accuracy of independently trained models of those sizes, reflecting 'effective capacity'.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.