Reproduce “Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Joshua Hill
The work introduces a "coverage model" to explain quantization damage in large language models as a first-order, saturating process, challenging assumptions of existing mixed-precision methods. This model enables more accurate loss prediction and bit-width allocation, achieving 17 –21% lower KL divergence compared to production baselines on MoE LLMs and enabling task survival at sub-4-bit effective bit-widths.
Key insights
- Quantization damage is predominantly first-order (85 –99% of variance) and is approximately ordinally additive after a strictly increasing transformation, with higher-order variance often reflecting loss scale curvature.
- A "saturation" phenomenon, where loss is bounded and marginal damage diminishes with increasing quantization, governs the damage structure and is accurately captured by the coverage model.
- Estimates built from isolated layer sensitivities significantly overstate actual quantization loss (by 71 –376%) because they are measured from an unquantized network rather than the deployment distribution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.