Reproduce “Sharp Minima Can Generalize For Deep Nets” on OpenResearch

About this paper

Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio

Research challenges the notion that "flat minima generalize better" in deep neural networks, demonstrating that common flatness definitions are unstable under reparametrization and inherent architectural symmetries, particularly in ReLU networks. It reveals that observationally equivalent models can correspond to arbitrarily sharp or infinitely flat minima, decoupling perceived flatness from true generalization.

Key insights

• Deep rectified networks possess inherent architectural symmetries and non-identifiability (e.g., "alpha"-scale transformations) that allow infinite sets of observationally equivalent parameters to compute the exact same function.
• The geometry of the loss landscape, including the perceived flatness or sharpness of minima, can be arbitrarily altered by choosing different valid parameterizations or reparametrizations, without changing the function computed by the network.
• Common flatness definitions are not intrinsic properties of the learned function but rather artifacts of the chosen parameter space and its coordinate system, rendering them unreliable as universal indicators of generalization.

Summary by alphaXiv.