Reproduce “AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization” on OpenResearch

About this paper

Wanqi Yang, Yuexiao Ma, Alexander Conzelmann, Xiawu Zheng, Michael W. Mahoney, T. Konstantin Rusch, Shiwei Liu

AlphaQ is a calibration-free bit allocation method for Mixture-of-Experts (MoE) quantization that estimates expert importance directly from model weight matrices using Heavy-Tailed Self-Regularization theory. This approach improves model performance and memory efficiency in MoE LLMs by avoiding the need for external calibration data, leading to up to 7.2x memory compression and competitive accuracy with full-precision models.

Key insights

• Spectral heavy-tailedness (quantified by PL_Alpha_Hill) of weight matrices provides a calibration-free indicator of layer importance in MoE models, correlating with better-trained and more informative features.
• Importance signals exhibit substantial heterogeneity across different MoE blocks and layers within experts, suggesting that a global, fine-grained bit allocation is more effective than uniform or block-wise assignments.
• An optimal bit allocation strategy emerges from scaling quantization noise by the derived alpha-based importance, minimizing total error under a global bit budget.

Summary by alphaXiv.