Reproduce “Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Meixuan He, Yuqing Liang, Jinlan Liu, Dongpo Xu
Convergence guarantees were established for vanilla Adam in non-convex settings under relaxed hyperparameter conditions, by researchers at Northeast Normal University. The work provided the first proof of non-ergodic convergence for Adam's last iterate to a stationary point and derived an O(1/K) non-ergodic rate for function values under the Polyak-Lojasiewicz condition.
Key insights
- Non-ergodic convergence (convergence of the last iterate) is a stronger and more practically meaningful result compared to ergodic convergence (convergence of an averaged iterate, minimum iterate, or uniformly sampled iterate).
- Vanilla Adam can achieve convergence under significantly more relaxed hyperparameter conditions (SC-Adam) than previously established, allowing for greater flexibility in practical tuning.
- The Polyak-Lojasiewicz (PL) condition provides a useful framework for analyzing faster non-ergodic convergence rates for function values even in non-convex settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.