Reproduce “Memory-Efficient LLM Pretraining via Minimalist Optimizer Design” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Athanasios Glentis, Jiaxiang Li, Andi Han, Mingyi Hong

SCALE is an optimizer for LLM pretraining that column-normalizes gradients across layers and stores first-order momentum only for the output layer. On LLaMA 7B trained on C4 for 19.7B tokens, it reports perplexity 12.59 and estimated weights-plus-optimizer memory of 13.74 GB, versus Muon’s 12.72 and 26.95 GB. These estimates exclude peak training memory; evidence is limited to one run each.

Key insights

  • Column-wise gradient normalization is crucial for stabilizing gradients, especially in the Language Modeling head, where it effectively mitigates extremely high absolute gradient values associated with certain frequent tokens.
  • Applying first-order momentum exclusively to the last layer significantly reduces its gradient variance and improves overall performance, while adding momentum to other layers offers negligible gains and increases memory usage.
  • A minimalist optimizer design, based on carefully chosen gradient normalization and selective momentum, can achieve Adam-level performance and stability with substantially lower memory requirements, suggesting redundancy in more complex adaptive optimizers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.