Reproduce “NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers” on OpenResearch

About this paper

Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, Shanghang Zhang

Researchers from Nanjing University, UC Berkeley, and Peking University developed NoisyQuant, a method that adds a theoretically justified uniform noisy bias to Vision Transformer activations to reshape their heavy-tailed distributions before post-training quantization. This approach significantly improves 6-bit quantization accuracy, bringing linear quantizers to parity with or exceeding more complex nonlinear methods and further enhancing state-of-the-art nonlinear quantizers across various ViT models.

Key insights

• Adding a uniform noisy bias to activations can theoretically and empirically reduce quantization error, especially for heavy-tailed distributions, by effectively smoothing the value distribution.
• Proactively altering the activation distribution itself, rather than solely adapting the quantizer to it, provides a powerful and novel strategy for improving PTQ performance.
• NoisyQuant offers a quantizer-agnostic, plug-and-play enhancement that consistently boosts the accuracy of both simple linear and complex nonlinear PTQ methods.

Summary by alphaXiv.