Reproduce “DeepMimic: Mentor-Student Unlabeled Data Based Training” on OpenResearch

About this paper

Itay Mosafi, Eli David, Nathan S. Netanyahu

DeepMimic, a training method from Bar-Ilan University researchers, enables deep neural networks to learn effectively from predominantly unlabeled data. It allows a student model to achieve performance comparable to a mentor by training on mentor-generated soft labels, frequently leading to simpler student architectures and faster inference times, for example, up to 35% faster on Tiny ImageNet with a simplified student model.

Key insights

• Knowledge can be effectively transferred from a mentor to a student using primarily unlabeled data and mentor-generated soft labels, yielding comparable performance even with architectural simplifications.
• Training with soft labels provides a richer, more generalized supervision signal than traditional one-hot encoding, leading to stronger regularization and more consistent student learning dynamics.
• The method facilitates significant architectural simplification in the student model while maintaining performance, which is beneficial for deployment on resource-constrained systems and mimicking black-box models.

Summary by alphaXiv.