Reproduce “Delayed Effects of Temporary Weight Decay in Muon Pretraining” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Md Rifat Arefin, Ravid Shwartz-Ziv, Ernie Chang, Chinnadhurai Sankar, Rylan Conway, Aristide Baratin, Irina Rish, Adithya Sagar and 1 more

The authors test temporary weight decay schedules in Muon-trained decoder-only Transformers and find that validation-loss rankings can change after decay ends. A schedule selected on a 162M model and transferred without retuning to 1.4B is 0.2616 nats worse at removal, then finishes 0.0275 nats better than no decay. These results concern validation loss in tested Muon regimes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.