Reproduce “Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eitan Gronich, Gal Vardi
The authors analyze how Adam’s learning-rate schedule shapes implicit margin bias in binary classification. In the fast regime, rescaled second-moment estimates converge to trajectory-dependent limits, inducing a diagonal preconditioner and a weighted-ℓ2 margin characterization. Under stated assumptions, directional limit points satisfy KKT conditions. Specified deep linear settings rule out a universal regularizer, not trajectory-dependent margin characterizations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.