Reproduce “Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kiwon Lee, Andrew N. Cheng, Courtney Paquette, Elliot Paquette
A rigorous analysis of mini-batch Stochastic Gradient Descent with Momentum (SGD+M) in high-dimensional settings reveals how mini-batch size dictates momentum's acceleration properties. The work introduces the Implicit Conditioning Ratio (ICR) to define two distinct batch size regimes: a large batch regime where momentum provides acceleration, and a small batch regime where it does not, with the theory validated on synthetic and real datasets like MNIST.
Key insights
- Momentum's acceleration in SGD+M is critically dependent on the batch size, specifically relative to the Implicit Conditioning Ratio (ICR).
- The discrete Volterra equation framework provides a powerful tool to analyze and predict the deterministic limits of stochastic optimization algorithms.
- Optimal batch sizing can be determined by the problem's Implicit Conditioning Ratio, beyond which increasing batch size provides no further acceleration benefit.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.