Reproduce “The Break-Even Point on Optimization Trajectories of Deep Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, Krzysztof Geras
The authors analyze how SGD learning rate and batch size shape training trajectories using a stability-based model and measurements of gradient covariance and loss curvature. Their model defines a break-even point, while experiments find that SimpleCNN on CIFAR-10 had maximum gradient-covariance eigenvalues of 0.68 and 3.30 at learning rates 0.1 and 0.01. These trajectory comparisons do not establish improved generalization or causality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.