Reproduce “Understanding and Overcoming the Challenges of Efficient Transformer Quantization” on OpenResearch

About this paper

Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort

Qualcomm AI Research identified that structured outliers within activation ranges, particularly in residual connections, cause significant accuracy drops when quantizing Transformer models to low bit-widths. They developed mixed-precision and a novel per-embedding-group post-training quantization, achieving near FP32 accuracy on the GLUE benchmark and enabling up to 8.85x memory reduction for these models.

Key insights

• Performance degradation in low-bit Transformer quantization is predominantly due to activation quantization, not weight quantization, with the 'residual sum after the FFN' being particularly problematic.
• High dynamic ranges in sensitive activation tensors contain 'structured outliers' in specific embedding dimensions, making it difficult for a single per-tensor scale to accurately represent the entire range.
• These structured outliers in FFN residual connections lead to skewed distributions in subsequent attention layers, causing most tokens to attend predominantly to special tokens like `[SEP]`.

Summary by alphaXiv.