Reproduce “Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning” on OpenResearch

About this paper

Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, Mingli Song

Researchers from Zhejiang University, Stevens Institute of Technology, and Alibaba Group developed a method to amalgamate knowledge from multiple deep neural networks with heterogeneous architectures into a single student model, relying solely on teacher outputs without human annotations. This approach achieves performance superior to individual teachers and even models trained with ground-truth on various image classification and face recognition benchmarks, facilitating efficient model reuse.

Key insights

• Knowledge amalgamation from heterogeneous architectures can be effectively achieved by projecting diverse features into a common latent space using adaptation layers and a shared extractor, rather than direct layer-wise imitation.
• Maximum Mean Discrepancy (MMD) is a robust statistical tool for aligning feature distributions from multiple teachers and the student within this common space, preventing trivial representations through a reconstruction loss.
• A single student model can learn to perform multiple tasks and even surpass the performance of individual specialized teachers and ground-truth trained models by intelligently synthesizing complementary knowledge from varied sources without needing new labels.

Summary by alphaXiv.