Reproduce “On the Expressive Power of Deep Neural Networks” on OpenResearch

About this paper

Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl-Dickstein

This research introduces "trajectory length" and "activation patterns" as new measures of neural network expressivity, demonstrating that functional complexity grows exponentially with network depth and revealing how Batch Normalization reduces this length, thereby stabilizing representations. A novel "Trajectory Regularization" method is also proposed, achieving comparable performance to Batch Normalization.

Key insights

• The functional complexity of deep neural networks, quantified by activation patterns and trajectory length, grows exponentially with network depth, providing a mechanistic explanation for the benefits of deeper architectures.
• Lower layers of a deep network are disproportionately more sensitive to weight perturbations and more critical for effective optimization, due to the exponential amplification of changes through successive layers.
• Batch Normalization regularizes deep networks by reducing and stabilizing trajectory length within the network's latent spaces, offering a quantifiable explanation for its empirical success.

Summary by alphaXiv.