Reproduce “Network Morphism” on OpenResearch

About this paper

Tao Wei, Changhu Wang, Yong Rui, Chang Wen Chen

Researchers from Microsoft Research Asia and the University at Buffalo developed Network Morphism, a framework that transforms a well-trained neural network into a new, often larger, architecture while mathematically guaranteeing the complete preservation of its original function. This method significantly reduces training time, achieving approximately a 15x speedup on ImageNet, and consistently produces models with enhanced generalization capabilities compared to training from scratch.

Key insights

• Network Morphism ensures complete function preservation during architectural changes, guaranteeing the child network starts with the same performance as its parent.
• Evolving networks through NetMorph demonstrates an 'internal regularization ability,' consistently achieving better accuracy than models of the same architecture trained directly from scratch.
• The parametric-activation function family broadens the applicability of network morphing to any continuous non-linear activation, overcoming limitations of prior methods restricted to idempotent functions.

Summary by alphaXiv.