Reproduce “Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise” on OpenResearch

About this paper

Xingyu Wang, Sewoong Oh, Chang-Han Rhee

A theoretical framework establishes that Stochastic Gradient Descent, when combined with gradient clipping and driven by heavy-tailed noise, actively eliminates sharp local minima from its long-term training trajectory. This mechanism leads to consistently improved generalization performance and the discovery of flatter minima in deep neural networks across image classification benchmarks.

Key insights

• The geometric "width" of a local minimum, quantified by the `l*` parameter (minimum clipped jumps to escape its attraction field), directly dictates the order of the first exit time from that minimum.
• When heavy-tailed noise is combined with gradient clipping, SGD undergoes a stronger regularization effect, leading to the effective *elimination* of sharp local minima from its long-term training trajectory.
• Gradient clipping is critical for the benefits of heavy-tailed noise; without it, injecting heavy-tailed noise can destabilize optimization and significantly degrade generalization performance.

Summary by alphaXiv.