Reproduce “Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cuong Pham, Tuan Hoang, Thanh-Toan Do

CMT-KD trains multiple quantized teachers and a low-bit image-classification student collaboratively, combining learned, layer-specific teacher features that are reused by teachers and distilled to the student. On CIFAR-100 ResNet18, its 2-bit student reaches 78.3% top-1 and 94.4% top-5, above an averaged-teacher student and full precision; this result is specific to the tested setting.

Key insights

  • Collaborative learning among multiple quantized teachers, dynamically weighted by learned importance factors, provides a richer and more suitable knowledge signal for a quantized student than static or full-precision teachers.
  • Mutual learning between teachers and students allows the student's learning progress to influence teacher refinement, leading to adaptive optimization and enhanced knowledge transfer.
  • The contributions of different teachers vary significantly across network layers and evolve during training, highlighting the benefit of dynamic, layer-dependent importance weighting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.