Reproduce “A Rod Flow Model for Adam at the Edge of Stability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Regis, Sinho Chewi
The authors develop Adam rod flow, a continuous-time model for Adam near the edge of stability. It represents parameter-and-momentum states with a rod and tracks the squared-gradient average separately. Once warmed into the steady-state edge-of-stability regime, it tracks the discrete trajectory’s center several orders of magnitude more closely than stable flow across three full-batch architectures; early or mini-batch training remain untested.
Key insights
- The rod flow framework is generalizable, accurately modeling the EoS dynamics of a broad class of optimizers including momentum-based and adaptive gradient methods.
- Adaptive preconditioners in methods like Adam enable "acceleration via regularization," where the preconditioner inflates to absorb excess sharpness, allowing the optimizer to tolerate higher actual sharpness without diverging.
- Adam's robustness at the EoS stems from the interplay of momentum, which raises the effective sharpness threshold, and adaptive preconditioning, which provides a mechanism to tolerate high sharpness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.