Reproduce “The Implicit Regularization of Momentum Gradient Descent with Early Stopping” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Li Wang, Yingcong Zhou, Zhiguo Fu
This paper establishes that Momentum Gradient Descent (MGD) with early stopping implicitly performs L2-regularization, specifically demonstrating a strong quantitative link between its continuous-time form and ridge regression. The work shows that MGD's implicit regularization is tighter and converges faster to ridge regression compared to standard Gradient Descent, providing a theoretical explanation for MGD's generalization benefits.
Key insights
- Momentum Gradient Flow (MGF) implicitly regularizes like ridge regression, with a specific calibration parameter of "lambda = 2/t^2" for early stopping time 't'.
- Momentum Gradient Descent (MGD) induces a tighter implicit regularization effect compared to standard Gradient Descent (GD), leading to significantly closer performance to explicit L2-regularization.
- The momentum term contributes to a faster decay of the implicit regularization strength, with an O(1/t^2) dependence compared to GD's O(1/t).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.