Reproduce “Effective Quantization of Muon Optimizer States” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Aman Gupta, Rafael Celente, Abhishek Shivanna, D.T. Braithwaite, Gregory Dexter, Shao Tang, Hiroto Udagawa, Daniel Silva and 2 more

Researchers at Nubank developed an 8-bit quantized version of the Muon optimizer that reduces optimizer state memory by up to 79% compared to full-precision AdamW, preserving model quality across large language model pre-training and fine-tuning, notably through Muon's compatibility with simple linear quantization.

Key insights

  • Muon's unique update mechanism, which primarily maintains a first-order momentum and lacks the second-moment normalization term, makes it robust to simple 8-bit linear quantization.
  • AdamW's instability under linear quantization stems from the second-moment accumulator in its denominator, which significantly magnifies quantization error, as proven by theoretical analysis.
  • Quantizing only Muon's states with linear 8-bit precision (Muon-8L/AdamW-32) maintains performance parity, while aggressive quantization of the AdamW component can introduce slight performance degradation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.