Reproduce “How to set AdamW's weight decay as you scale model and dataset size” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xi Wang, Laurence Aitchison

The authors study AdamW weight-decay transfer, interpreting decoupled decay as setting an approximate memory timescale for recent updates. Their experiments find that the preferred epoch-based timescale is roughly stable as dataset size grows, while scaling decay with width under μP learning-rate scaling stabilizes the best base learning rate. Evidence is limited to tested high-epoch settings.

Key insights

  • AdamW's weight updates can be accurately reinterpreted as an Exponential Moving Average (EMA) of recent updates, introducing an 'EMA timescale' (τ_iter = 1/(ηλ)) as a key hyperparameter.
  • The optimal EMA timescale, measured in epochs (τ_epoch = 1/(ηλ N/B)), remains stable across varying model sizes and dataset sizes, providing a theoretical basis for principled weight decay scaling.
  • The magnitude of learned weights scales inversely with weight decay (W_ij ~ 1/λ), implying that proper weight decay scaling also ensures appropriate scaling of learned weights according to μP principles.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.