Reproduce “Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps” on OpenResearch

About this paper

Subham Singh, Ashutosh Mishra, Subha Raut

This study offers a theoretical framework that explains how gradient noise structure and optimizer normalization together determine whether a learning-rate cooldown phase benefits final training loss, demonstrating that Stochastic Gradient Descent self-anneals under multiplicative noise while normalized optimizers require cooldown due to fixed update magnitudes. The work provides a principled diagnostic for predicting schedule preferences based on measurable gradient noise characteristics.

Key insights

• SGD inherently 'self-anneals' when gradient noise is purely multiplicative and vanishes at the optimum, as its step size naturally decreases with gradient magnitude, thus making cooldown unnecessary.
• Normalized optimizers (e.g., signSGD, Normalized SGD) cannot self-anneal under multiplicative noise because their update magnitude is fixed or less sensitive to gradient magnitude, leading to an inherent last-iterate loss floor of order "u0398("u03b7"u00b2)".
• The presence of an additive noise component (noise floor) makes cooldown beneficial for all optimizers, as reducing the learning rate directly lowers this noise floor, improving convergence.

Summary by alphaXiv.