Reproduce “Embedding Compression for Teacher-to-Student Knowledge Transfer” on OpenResearch

About this paper

Yiwei Ding, Alexander Lerch

Researchers at the Music Informatics Group, Georgia Institute of Technology, developed an embedding compression module for the "Embeddings as Teachers" (EAsT) knowledge transfer framework. This module filters task-irrelevant information from teacher embeddings, enabling lightweight student models to achieve improved classification performance, particularly with unsupervised teacher models and enhanced generalizability across various audio tasks.

Key insights

• Unsupervised pre-trained embeddings contain a greater amount of task-irrelevant knowledge, making them benefit most significantly from the proposed embedding compression during knowledge transfer.
• A simple linear projection is sufficient for effective teacher embedding compression, outperforming more complex transformations by preventing overfitting to the specific target task.
• The embedding compression process effectively prunes irrelevant knowledge without diminishing the student model's generalizability, suggesting it focuses on removing redundant or distracting features.

Summary by alphaXiv.