Reproduce “The Marginal Value of Adaptive Gradient Methods in Machine Learning” on OpenResearch

About this paper

Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht

This research from UC Berkeley and TTIC demonstrates that widely used adaptive gradient methods (e.g., Adam, AdaGrad) often lead to solutions that generalize worse than those found by non-adaptive methods (e.g., SGD with momentum) in deep learning. The paper provides both theoretical analysis in simplified overparameterized settings and comprehensive empirical evidence across various deep neural network architectures and tasks.

Key insights

• In overparameterized settings, non-adaptive methods like SGD converge to minimum Euclidean norm solutions, associated with good generalization, while adaptive methods can converge to solutions with different properties, such as low L-infinity norm characteristics, which may generalize poorly.
• Adaptive methods can give undue influence to 'spurious' features that appear useful for training data but harm out-of-sample generalization, leading to severe overfitting in theoretical examples.
• Despite faster initial training progress, adaptive methods often plateau on development/test performance, whereas non-adaptive methods, though sometimes slower initially, continue to improve and achieve better final generalization.

Summary by alphaXiv.