Reproduce “MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

Microsoft Research developed MINILMv2, a method for compressing large Transformer models through multi-head self-attention relation distillation. This approach enables student models to efficiently mimic teacher models, even with differing numbers of attention heads, and consistently yields improved performance and efficiency, for instance, a 6-layer student distilled from BERT_LARGE achieved an 81.7 GLUE score, surpassing BERT_BASE's 81.5 while being 2.0x faster.

Key insights

  • Distilling fine-grained self-attention relations offers a more robust and effective knowledge transfer mechanism than traditional attention map distillation, improving student model performance.
  • The concept of "relation heads" effectively decouples the distillation process from the specific number of attention heads in the teacher and student, providing greater architectural flexibility for compressed models.
  • For large teacher models, knowledge contained in "upper middle layers" can be more beneficial for student learning and generalization than knowledge from the final layer.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.