Reproduce “Scaling and Transferability of Annealing Strategies in Large Language Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siqi Wang, Zhengyu Chen, Teng Xiao, Zheqi Lv, Jinluan Yang, Xunliang Cai, Jingang Wang, Xiaomeng Li
The authors model training loss by update steps to estimate the decay fraction in warmup-stable-decay (WSD) learning-rate schedules. Their framework combines cumulative learning-rate area, an Adam-style term for decay behavior, and model size. They report that the estimated optimum transfers across sizes within Dense and MoE families; in one 1B MoE setting, final-loss error is within 0.003 across training and validation sets. Transfer between families is not established.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.