Reproduce “Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves, Aman Gupta, Hiroto Udagawa, Abhishek Shivanna and 2 more

ZIP-SR and ZE-EDEN compress AdamW’s moment states to four bits while choosing second-moment quantization to account for its effect on adaptive updates. ZIP-SR rounds in preconditioner space; ZE-EDEN calibrates a zero-excluding codebook. In GPT-style 1.4B pretraining, ZIP-SR reduces the mean paired validation-loss gap versus TorchAO by 70.1%; downstream scores show no consistent advantage for either recipe.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.