Reproduce “The Road Less Scheduled” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky
This research introduces Schedule-Free learning, a novel optimization approach that eliminates the need for pre-specified learning rate schedules by dynamically combining iterate averaging with a unique form of momentum. The method, developed by researchers from Meta FAIR, Google Research, Samsung AI Center, Princeton University, and Boston University, matches or exceeds the performance of heavily-tuned schedule-based optimizers across 28 diverse benchmarks and was the core algorithm behind the winning entry in the MLCommons 2024 AlgoPerf Algorithmic Efficiency Challenge Self-Tuning track.
Key insights
- Iterate averaging, when combined with a specific form of momentum (interpolating between the last iterate and the average iterate for gradient evaluation), can achieve both strong theoretical guarantees and superior empirical performance, effectively bridging the theory-practice gap related to Polyak-Ruppert averaging.
- Learning rate scheduling and iterate averaging can be unified under a single theoretical framework, demonstrating a deeper connection between these techniques and enabling truly "schedule-free" optimization.
- Schedule-Free methods facilitate the use of larger base learning rates, which contributes to faster convergence, and prevent phenomena like gradient norm collapse, maintaining effective learning throughout training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.