Reproduce “High-dimensional Limit of SGD for Diagonal Linear Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel, Dmitriy Drusvyatskiy
The study from the University of Washington, McGill University, and Google DeepMind establishes a continuous-time SDE and a deterministic PDE to precisely model SGD dynamics for diagonal linear networks in the high-dimensional, fixed-stepsize limit. This framework allows for the deterministic prediction of observable statistics like risk and curvature, and shows global linear convergence to zero risk with high probability for specific network configurations.
Key insights
- High dimensionality serves as a primary simplifying mechanism, enabling the derivation of accurate continuous-time SDE and deterministic PDE descriptions for SGD even with fixed, non-vanishing stepsizes.
- A broad class of observable statistics, including empirical risk and curvature, concentrate around deterministic functions in the high-dimensional limit, allowing for their precise prediction without Monte Carlo simulations.
- For certain network configurations, the derived dynamics guarantee global linear convergence to zero risk with high probability, offering strong theoretical guarantees on optimization outcomes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.