Reproduce “Knowledge distillation through geometry-aware representational alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Prajjwal Bhattarai, Mohammad Amjad, Dmytro Zhylko, Tuka Alhanai
Researchers from New York University Abu Dhabi introduced Procrustes distance and the Frobenius norm of the Feature Gram Matrix Difference (FGMD) as geometry-aware distillation losses for transferring knowledge between deep learning models. These methods achieved state-of-the-art performance on the CoLA task (56.0 MCC) for BERT distillation and provided up to a 2 percentage point improvement in Rogue-L scores for OPT instruction-following tasks compared to previous logit-based methods.
Key insights
- Centered Kernel Alignment (CKA) and projection-based methods can yield arbitrarily small loss values without guaranteeing the preservation of the underlying geometric structure of feature representations.
- Procrustes distance and FGMD directly align the intrinsic geometric properties of feature spaces, offering a superior mechanism for transferring deep structural knowledge compared to traditional methods.
- Effective knowledge distillation for task performance requires a combination of both geometry-aware feature alignment and output-level information (e.g., KL divergence or cross-entropy loss).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.