Reproduce “Measuring the Effects of Data Parallelism on Neural Network Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl

Researchers at Google Brain conducted an extensive empirical study across 35 diverse neural network workloads to characterize how increasing data parallelism (batch size) affects training speed and model quality. The study revealed a universal three-regime relationship between batch size and training steps, demonstrating that proper, independent metaparameter tuning prevents inherent degradation in solution quality with large batches.

Key insights

  • A universal, S-shaped relationship exists between batch size and the number of training steps, characterized by regions of perfect scaling, diminishing returns, and a maximum useful batch size.
  • Momentum-based optimizers significantly extend the perfect scaling region to larger batch sizes, leading to substantial reductions in total training steps compared to plain SGD.
  • Reported degradations in solution quality with large batches are primarily artifacts of suboptimal metaparameter tuning or specific computational budget assumptions, rather than an inherent consequence of increased batch size.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.