Reproduce “The Ephemeral Cost of Switching Optimizers Across LLM Training Stages” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma, Rameswar Panda, Paul Pu Liang
The authors test whether switching between AdamW and Muon from pretraining to task-general supervised fine-tuning (SFT) harms downstream performance. They vary each optimizer’s share independently in 1.2B Llama-3 models and assess six benchmark scores. For AdamW-pretrained models, Muon SFT averages 17.2 versus 15.8 for AdamW SFT; matching reduces measured forgetting, but yields no consistent benchmark advantage in these tests.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.