Reproduce “Fantastic Pretraining Optimizers and Where to Find Them” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang
The authors compare 11 language-model optimizers, tuning each and evaluating final validation loss across four model sizes and multiple token budgets. They estimate token efficiency by asking how many tokens AdamW would need to match each alternative’s loss, not by measuring elapsed time. At 1.2B parameters and 8× the Chinchilla-optimal budget, Muon, Soap, and NAdamW each yield about 1.1× estimated speedup; models stop at 1.2B.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.