Reproduce “Optimizer-dependent training dynamics converge to the same one-third optimal data scaling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyunseok Lee, Mihir Basil, Yizhou Liu, Jeff Gore
The authors study how optimizer choice affects training dynamics and optimally tuned loss per sample in an online, single-layer teacher–student classifier. They compare seven optimizers and decompose loss into radial and tangential components. Although the components and learning-rate optima differ, pooled SGD and Adam envelopes have exponent 0.341; this supports a common one-third rate in the toy model, not Transformers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.