Reproduce “Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingi Ji, Byeongho Heo, Sungrae Park

Researchers from KAIST and NAVER developed Attention-based Feature Distillation (AFD), a method that uses attention mechanisms to automatically learn dynamic links between teacher and student network features. AFD achieves superior performance in model compression and transfer learning tasks compared to existing methods, showing up to 0.7% accuracy improvement on CIFAR-100 and enabling efficient application on datasets like ImageNet.

Key insights

  • AFD's attention mechanism adaptively learns feature links based on architectural differences, enabling effective knowledge transfer even between disparate teacher-student networks.
  • The method significantly reduces computational overhead compared to prior automatic linking approaches by co-training the attention network simultaneously with the student, avoiding expensive meta-learning steps.
  • Qualitative analysis reveals that AFD can spread high-level teacher knowledge across multiple student layers, optimizing information flow and demonstrating flexible knowledge utilization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.