Reproduce “On the Selection of Initialization and Activation Function for Deep Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Soufiane Hayou, Arnaud Doucet, Judith Rousseau
The authors analyze how activations and initialization affect information propagation in networks. In an infinite-width Gaussian-process model, they show ReLU correlations approach one at the edge of chaos at a polynomial rate, while Swish can be tuned to slow convergence. On MNIST, a width-60, depth-40 network achieves 97.14% test accuracy with Swish and 91.45% with ReLU, without isolating initialization.
Key insights
- Initializing ReLU-like networks at the "edge of chaos" transitions the correlation decay rate from exponential to polynomial with respect to network depth, improving information propagation.
- Even at the "edge of chaos", ReLU-like activations ultimately lead to degenerate kernels, where correlations converge to 1, indicating an inherent limitation for very deep networks.
- A new class of activation functions, exemplified by Swish, can be theoretically shown to maintain correlations near identity, enabling more robust and diverse information flow across arbitrarily deep layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.