Reproduce “Transforms for LLM Quantization: The Great Inversion and Format Co-Design” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ehsan Jokar
The authors synthesize theory and prior evidence on transforms for low-bit language-model quantization, identifying a “Great Inversion”: concentration can help flexible-bit classical coding, while flattening can help uniform integer grids with shared scales. They survey 200 works and classify 43 methods. In matched Llama-2-7B W4A4 testing, QuaRot reaches 8.56 perplexity versus SmoothQuant’s 83.12; this is not a universal transform optimum.
Key insights
- "The Great Inversion" principle demonstrates that classical allocation-flexible coding rewards energy concentration, while modern grouped shared-scale quantization rewards energy flattening within groups to minimize outlier impact.
- The "crest factor" (ratio of maximum to RMS magnitude) directly determines quantization error in AbsMax shared-scale systems, making transforms that reduce it crucial for low-bit accuracy.
- Global orthogonal transforms, like Hadamards, applied to the residual stream in RMSNorm-based architectures, can be folded into surrounding weights at zero inference cost due to computational invariance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.