Reproduce “ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qizhen Lan, Qing Tian

ACAM-KD introduces an adaptive and cooperative attention masking framework for knowledge distillation, enhancing lightweight student models for dense visual prediction tasks. The method achieves up to 1.4 mAP improvement over state-of-the-art KD methods on COCO2017 object detection and up to 3.09% mIoU gain on Cityscapes semantic segmentation, by dynamically tailoring knowledge transfer to the student's evolving learning state.

Key insights

  • Dynamic and cooperative interaction between teacher and student, where the teacher guides attention based on the student's evolving features, leads to more effective knowledge transfer than static or purely teacher-driven approaches.
  • Jointly optimizing both spatial and channel-wise feature importance through adaptive masking is crucial for comprehensive knowledge transfer in dense visual prediction.
  • A diversity loss is essential for ensuring learnable masks capture distinct and complementary feature patterns, preventing degeneracy and fostering richer knowledge acquisition.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.