Reproduce “On the Impact of the Activation Function on Deep Neural Networks Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Soufiane Hayou, Arnaud Doucet, Judith Rousseau

The authors study initialization and activation choices for signal propagation and learning in deep fully connected networks. EOC initialization sets weights and biases at a critical boundary before training, while activation choice affects how input distinctions persist. In a 200-layer, width-300 ELU network trained on MNIST with SGD, EOC reached 97.62 ± 0.21% test accuracy after 100 epochs, versus 93.41 ± 0.30% with EOC plus batch normalization and 10.14 ± 0.51% with ordered-phase initialization; this finding is limited to that tested setting.

Key insights

  • Initializing on the EOC enhances information propagation and accelerates training, causing ReLU networks to propagate correlations similarly to residual networks with an O(1/l^2) convergence rate.
  • Smooth activation functions (e.g., Tanh, ELU) facilitate deeper signal propagation with a faster O(1/l) correlation convergence rate on the EOC, theoretically justifying their empirical advantages.
  • The parameter eta_q on the EOC controls both forward information flow and backward gradient stability, suggesting an optimal eta_q approx L (network depth) for selecting sigma_b to maximize information flow.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.