Reproduce “Identifying Sensitive Weights via Post-quantization Integral” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuezhou Hu, Weiyu Huang, Zichen Liang, Chang Chen, Jintao Zhang, Jun Zhu, Jianfei Chen
A new Post-quantization Integral (PQI) metric was developed to accurately measure the element-wise impact of quantization on Large Language Model loss, overcoming the limitations of traditional sensitivity metrics. The accompanying ReQuant framework, which employs PQI for self-adaptive outlier selection and step-wise significant weight detachment, consistently improves the perplexity of 3-bit and 4-bit quantized Llama 3.2 models, including enabling functional performance on tasks like MATH generation where baselines failed.
Key insights
- Traditional gradient and Hessian-based sensitivity metrics are profoundly inaccurate for LLMs, severely underestimating actual loss changes (by orders of magnitude) and sometimes predicting incorrect trends due to the “small convergence radius” of Taylor approximations.
- The Post-quantization Integral (PQI) accurately predicts total loss change by integrating the gradient along the entire path from original to quantized weights, providing fine-grained, element-wise sensitivity information.
- Leveraging accurate sensitivity metrics like PQI through adaptive outlier selection and step-wise high-precision detachment significantly enhances the performance of low-bit quantized LLMs, making them viable for challenging tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.