Reproduce “Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade
The authors test horizon-free learning-rate schedules paired with online weight averaging for open-ended language-model pretraining. They compare constant and shifted inverse-square-root rates with cosine schedules tuned separately for each training horizon, using 150M- and 300M-parameter models. A single averaged run stays near the cosine validation-loss envelope, with results limited to these models and setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.