Reproduce “Critical Points of Random Neural Networks” on OpenResearch

About this paper

Simmaco Di Lillo

This paper quantifies the expected number of critical points of random deep neural networks at initialization in the infinite-width limit. It identifies three distinct asymptotic growth regimes (constant, polynomial, exponential) for the number of critical points, which are governed by a spectral parameter derived from the activation function.

Key insights

• The Covariance Regularity Index (CRI > 2) defines a critical smoothness condition for applying classical Kac-Rice theory to analyze critical points in wide neural networks.
• The functional complexity of random neural networks, measured by the expected number of critical points, aligns with previously identified spectral complexity regimes based on a single parameter, the first derivative of the covariance function at 1, ", "'(1).
• Irregular activation functions like ReLU, which do not meet the CRI > 2 condition, exhibit distinct behavior in numerical simulations, suggesting a potentially infinite number of critical points and a fractal-like geometry in the continuous limit.

Summary by alphaXiv.