Reproduce “Scaling Laws For Mixed Quantization” on OpenResearch

About this paper

Zeyu Cao, Boyang Gu, Cheng Zhang, Pedro Gimenes, Jianqiao Lu, Jianyi Cheng, Xitong Gao, Yiren Zhao

Unified scaling laws for mixed quantization in Large Language Models (LLMs) are proposed and empirically validated, characterizing how model loss degenerates with model size, quantization ratio, and block size. The work shows that larger models can accommodate a progressively higher proportion of low-precision parameters and that very small quantization block sizes offer diminishing returns for large models, which has implications for efficient LLM deployment and hardware design.

Key insights

• Larger LLMs can accommodate a progressively higher quantization ratio (Q_r) for a fixed loss budget, making mixed quantization more effective as models scale.
• For sufficiently large models (e.g., >50B parameters), very small quantization block sizes (Q_b) yield diminishing returns, suggesting that fine-grained hardware support may not be strictly necessary for optimal performance-cost trade-offs.
• The unified scaling law provides a principled, predictive framework for designing and optimizing mixed quantization strategies, moving beyond heuristic approaches.

Summary by alphaXiv.