Reproduce “BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models” on OpenResearch

About this paper

Liulu He, Shenli Zheng, Karwei Sun, Yijiang Liu, Yufei Zhao, Chongkang Tan, Huanrui Yang, Yuan Du and 1 more

BASE-Q introduces a post-training quantization framework for Large Language Models, enhancing rotational quantization with blockwise bias correction and asymmetric scaling. This method achieves state-of-the-art accuracy for W4A4KV4 configurations, reducing the accuracy gap to full precision by up to 50.5% compared to prior methods, while enabling the quantization of 70B models on a single A800 GPU.

Key insights

• Rotational quantization methods, while effective at outlier suppression, often fail to align channel means, leaving a substantial component of rounding error unaddressed.
• Transforming activations to a Gaussian-like distribution via rotation exacerbates clipping errors, as significant energy in the tails is lost when applying standard clipping thresholds.
• Complex, learned global rotations can be largely redundant when blockwise bias correction is applied, suggesting that localized error mitigation is more efficient than global optimization for certain error types.

Summary by alphaXiv.