Reproduce “The loss landscape of overparameterized neural networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Y Cooper
Research from the Institute for Advanced Study mathematically characterizes the loss landscape of overparameterized neural networks, proving that their global minima form an n-d dimensional smooth manifold and that the Hessian at these minima generically exhibits n-d zero eigenvalues and d positive eigenvalues. This work provides rigorous theoretical backing for empirical observations of "flatness" in deep learning loss landscapes.
Key insights
- The set of global minima for overparameterized neural networks forms a high-dimensional smooth manifold, not isolated points, simplifying the optimization problem.
- Empirical observations of "flatness" in the loss landscape are mathematically confirmed by the Hessian possessing n-d zero eigenvalues at global minima.
- Sufficiently wide feedforward neural networks, utilizing rectified smooth activation functions, are proven to perfectly memorize training data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.