Reproduce “Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jia-Nan Wang, Zixun Huang, Kairui Li, Zilin Wang, Lei Wu
The authors analyze momentum in fixed-budget noisy power-law kernel regression. They track stability and risk to tune learning rates and momentum as batch size grows. Polyak preserves the small-batch data-scaling exponent over a wider range; Nesterov improves it in a large-batch regime. For s=0.3, q=4, b=0.4, predicted exponents are 0.18, about 0.231, and 0.30; this is theoretical, not neural-network evidence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.