Reproduce “On the curvature of the loss landscape” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alison Pouplin, Hrittik Roy, Sidak Pal Singh, Georgios Arvanitidis
A rigorous differential geometric framework quantifies the curvature of deep learning loss landscapes, establishing that smaller scalar curvature corresponds to increased robustness to weight perturbations and improved generalization. This work formalizes the flatness hypothesis, showing that scalar curvature offers a more comprehensive measure of "flatness" than the Hessian's trace and helps reconcile it with reparameterization invariance.
Key insights
- Scalar curvature, derived from a Riemannian manifold framework, provides a comprehensive and principled measure of the loss landscape's local geometry, distinct from and often superior to the trace of the Hessian.
- Smaller scalar curvature directly correlates with greater robustness to weight perturbations, establishing a formal link between intrinsic geometric properties and generalization performance.
- The proposed framework reconciles the flatness hypothesis with reparameterization invariance by asserting that relative curvature comparisons within a specific parameterization are meaningful for generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.