Reproduce “Understanding Gradient Descent on Edge of Stability in Deep Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sanjeev Arora, Zhiyuan Li, Abhishek Panigrahi
This research offers a mathematical framework to explain the "Edge of Stability" phenomenon in deep learning, showing how Gradient Descent, with finite learning rates, implicitly regularizes solutions toward flatter minima by driving a deterministic flow on the manifold of minimizers.
Key insights
- GD can provably enter and operate within the EoS phase for specific loss landscape structures and learning rate schemes, particularly when near a manifold of minimizers.
- A novel implicit regularization mechanism, driven by finite learning rates, emerges in the EoS phase, where GD iterates evolve along a deterministic flow on the minimizer manifold that minimizes solution sharpness (i.e., seeks flatter minima).
- This implicit regularization is fundamentally dependent on finite learning rates, distinguishing it from many prior implicit bias results that rely on infinitesimal updates or stochastic noise.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.