Reproduce “Toward Understanding Why Adam Converges Faster Than SGD for Transformers” on OpenResearch

About this paper

Yan Pan, Yuanzhi Li

Researchers at Carnegie Mellon University identify directional sharpness as a key factor explaining why adaptive optimizers like Adam outperform SGD in training Transformer models. Their work demonstrates that coordinate-wise clipping can universally reduce this sharpness, accelerating convergence for various optimizers on Transformer-based tasks.

Key insights

• SGD exhibits significantly higher directional sharpness in its update directions compared to adaptive algorithms when training Transformers, which severely limits its potential step sizes and convergence speed.
• This high directional sharpness in SGD is primarily caused by a highly imbalanced distribution of gradient magnitudes, where a small fraction of coordinates has disproportionately large gradients.
• Adaptive optimizers implicitly mitigate this issue through their coordinate-wise scaling, while coordinate-wise clipping explicitly achieves a similar effect, leading to a flatter local loss landscape along the update direction.

Summary by alphaXiv.