Reproduce “Adaptive Gradient Methods at the Edge of Stability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze and 3 more
Adaptive optimizers exhibit an “Adaptive Edge of Stability” in which the largest eigenvalue of the preconditioned Hessian approaches the local stability limit predicted by a frozen-preconditioner analysis. Across Adam and related methods, architectures, datasets, and large-batch settings, the optimizer can remain stable while raw Hessian sharpness continues to increase because gradient-driven preconditioner updates compensate for high curvature.
Key insights
- Full-batch Adam typically drives preconditioned sharpness toward its stability boundary, while related methods show analogous behavior around thresholds determined by their momentum structure.
- Unlike non-adaptive methods, Adam can remain near this boundary while raw Hessian sharpness rises: large-gradient oscillations increase the second-moment accumulator, which reduces effective curvature without necessarily moving to a flatter region.
- Adaptive preconditioner dynamics therefore provide a mechanism for reaching sharper solutions, with the response depending on learning rate, β₂, batch size, and the optimizer’s momentum formulation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.