Reproduce “Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference” on OpenResearch

About this paper

Junyi Luo, Xinting Jiang, Tai-Hao Wen, Ruichen Qi, Minxing Chu, Hongyi Wu, Gregory Kielian, Ben Laurie and 4 more

AdaMX, a family of operand-specialized quantization formats, enables efficient low-bit Large Language Model (LLM) inference by dynamically selecting precision-recovery schemes for each tensor block and tailoring representation strategies for weights and activations. This approach, which repurposes underutilized bits in standard MX exponent fields, achieves up to 72% recovery of the perplexity gap between MXFP4 and FP16 with a hardware overhead of merely 3.2% in area and 3.5% in power for the systolic array.

Key insights

• Quantization heterogeneity is a critical factor limiting the accuracy of low-bit LLM quantization; an adaptive, per-block approach to precision recovery significantly improves model performance.
• The 8-bit exponent field in standard MX formats contains sufficient redundancy to embed sophisticated metadata for dynamic precision recovery without increasing the overall memory footprint of the quantized tensor.
• Operand-specialized quantization strategies (different formats and encoding processes for weights vs. activations) are crucial for optimizing both accuracy and hardware efficiency, given their distinct statistical properties and processing constraints.

Summary by alphaXiv.