Reproduce “Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lei Wu, Zhanxing Zhu, Weinan E

This paper explores why deep neural networks generalize effectively despite over-parameterization, proposing that the inherent geometry of the loss landscape guides optimizers to effective solutions. The study demonstrates that well-generalizing minima reside in large, flat basins of attraction, which are overwhelmingly more accessible from random initialization than sharp, poorly generalizing minima.

Key insights

  • Well-generalizing solutions correspond to flat minima in the loss landscape, characterized by smaller Hessian norms, while poorly generalizing solutions reside in sharp, narrow valleys with larger Hessian eigenvalues.
  • The basins of attraction for well-generalizing minima are volumetrically dominant in the high-dimensional parameter space compared to those of poorly generalizing minima.
  • The inherent structural properties of the loss landscape, rather than primarily SGD stochasticity or explicit regularization, fundamentally guide the optimization process towards solutions that generalize effectively.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.