Reproduce “Initialization of ReLUs for Dynamical Isometry” on OpenResearch

About this paper

Rebekka Burkholz, Alina Dubatovka

This research develops an exact theoretical framework for signal propagation in finite-width ReLU networks at initialization, revealing that standard initialization leads to signal alignment issues in deeper layers. It proposes a parameter-sharing scheme with an orthogonal submatrix structure that achieves dynamical isometry, demonstrating improved and stable classification accuracy for deep ReLU networks on MNIST and CIFAR-10 datasets compared to standard He initialization.

Key insights

• The exact output signal distribution in finite-width networks depends solely on the scalar products of input signals, extending the understanding beyond mean field assumptions.
• Standard ReLU initialization leads to an increase in average cosine similarity between distinct signals across layers, causing them to become indistinguishable (signal alignment problem) and limiting deep network trainability.
• The proposed parameter-sharing weight structure allows for perfect dynamical isometry in ReLU networks at initialization, overcoming the signal alignment issue by effectively propagating both positive and negative signal components.

Summary by alphaXiv.