Reproduce “The Emergence of Spectral Universality in Deep Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

Researchers from Google Brain and Stanford University developed a theoretical framework using free probability theory to analyze the full singular value spectrum of deep network Jacobians, identifying conditions under which universal limiting spectral distributions emerge in the deep limit. Their work demonstrates that orthogonal weight initialization is crucial for maintaining stable Jacobian conditioning and preventing variance growth with depth for certain nonlinearities.

Key insights

  • Even when the mean squared singular value is stable (at the critical line, \u00ce\u00a7=1), the variance of the Jacobian's squared singular values generally grows linearly with depth, indicating increasing ill-conditioning.
  • Orthogonal weights are a necessary prerequisite for the existence of stable, universal limiting spectral distributions in deep networks, preventing unbounded variance growth with depth.
  • Two universality classes (Bernoulli and Smooth) emerge for the Jacobian spectrum in the deep limit under specific scaling conditions, depending on the behavior of the nonlinearity's derivative.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.