Reproduce “ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin and 8 more
Researchers at Meta AI introduced ParetoQ, a unified framework for systematically evaluating and optimizing large language model quantization across extremely low bit-widths, identifying 2-bit and 3-bit quantization as offering superior accuracy-efficiency trade-offs compared to 4-bit approaches while achieving state-of-the-art performance for each level.
Key insights
- QAT fine-tuning with approximately 10% of the total training budget, following 90% full-precision pretraining, consistently yields optimal performance for low-bit quantization.
- Lower bit-widths (<=2-bit) require significantly more QAT fine-tuning tokens (~30B) due to a 'reconstruction' learning behavior, while higher bit-widths (>2-bit) require fewer tokens (~10B) due to 'compensation' learning.
- The choice of quantization function is highly sensitive and bit-width specific, with learnable scales outperforming fixed ranges, and symmetric quantization grids (like SEQ) being crucial for 1.58-bit and 2-bit models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.