Reproduce “Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wojciech Tarnowski, Piotr Warchoł, Stanisław Jastrzębski, Jacek Tabor, Maciej A. Nowak

Researchers at Jagiellonian University demonstrated that Residual Networks achieve dynamical isometry universally for any activation function, characterized by a single "effective cumulant" parameter. This enables a standardized approach to network initialization and facilitates meaningful comparisons of activation function performance.

Key insights

  • The singular value spectrum of the input-output Jacobian in ResNets is universally described by a single effective cumulant parameter, independent of the specific activation function.
  • Dynamical isometry can be achieved in ResNets for virtually any activation function by appropriately scaling the weight variance based on its calculated effective cumulant.
  • Initializing ResNets with a constant effective cumulant provides a standardized method for comparing activation functions, removing initialization as a confounding variable in performance studies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.