Reproduce “Exponential expressivity in deep neural networks through transient chaos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, Surya Ganguli
This research employs Riemannian geometry and dynamical mean field theory to provide a general framework for deep neural network expressivity. It reveals that deep random networks, driven by "transient chaos," enable exponential growth in the geometric complexity of input manifolds and decision boundaries with increasing depth.
Key insights
- The phenomenon of "transient chaos," where representations of nearby input points separate with depth, is the fundamental mechanism driving exponential expressivity in deep networks.
- Deep networks continuously generate and maintain high levels of extrinsic curvature through their layers, allowing for the creation of new geometric complexity rather than merely stretching existing features.
- The exponential growth in global manifold curvature (Grassmannian length) and local decision boundary curvature is a distinct advantage of depth, which shallow networks cannot efficiently replicate even with increased width.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.