Reproduce “Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves, Aman Gupta, Hiroto Udagawa, Abhishek Shivanna and 2 more
ZIP-SR and ZE-EDEN compress AdamW’s moment states to four bits while choosing second-moment quantization to account for its effect on adaptive updates. ZIP-SR rounds in preconditioner space; ZE-EDEN calibrates a zero-excluding codebook. In GPT-style 1.4B pretraining, ZIP-SR reduces the mean paired validation-loss gap versus TorchAO by 70.1%; downstream scores show no consistent advantage for either recipe.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.