Reproduce “Revisiting the Noise Model of Stochastic Gradient Descent” on OpenResearch

About this paper

Barak Battash, Ofir Lindenbaum

Research from Bar Ilan University demonstrates that stochastic gradient noise (SGN) in deep neural networks follows heavy-tailed Lévy alpha-stable distributions, not Gaussian, and importantly, these noise characteristics vary significantly across different network parameters. The study introduces a new theoretical framework using Lévy-driven stochastic differential equations to model this parameter-specific noise, providing insights into learning rate decay and escape dynamics from local minima.

Key insights

• Stochastic gradient noise in deep learning models exhibits heavy-tailed characteristics, better described by Lévy alpha-stable distributions than Gaussian.
• Noise properties (specifically the stability parameter α) are heterogeneous across different parameters within the same neural network, challenging the uniform noise assumption.
• Learning rate decay not only reduces step size but also directly attenuates the amplitude and scale of stochastic gradient noise.

Summary by alphaXiv.