Reproduce “Scaling Laws for Post Training Quantized Large Language Models” on OpenResearch

About this paper

Zifei Xu, Alexander Lan, Wanzin Yazar, Tristan Webb, Sayeh Sharify, Xin Wang

Researchers from d-Matrix and Yale University established empirical scaling laws for post-training quantized Large Language Models, demonstrating that factors like model size, loss landscape properties, and quantization formats predictably influence performance. They developed a statistical model that accurately predicts the quality of quantized LLMs, aiming to transform PTQ from a trial-and-error process into a predictable engineering discipline for efficient deployment.

Key insights

• The local loss landscape of pre-trained LLMs scales predictably with model size, with larger models exhibiting flatter landscapes and thus greater robustness to quantization.
• Microscaling (MX) quantization errors (SQNR) are largely invariant across different LLM families and model sizes, offering high predictability in contrast to traditional integer quantization.
• The effectiveness of GPTQ in reducing NLL loss is systematic but variable, yielding substantial improvements within a 'narrow window in SNR' related to the model's loss landscape.

Summary by alphaXiv.