Reproduce “An Analysis on Quantizing Diffusion Transformers” on OpenResearch

About this paper

Yuewei Yang, Jialiang Wang, Xiaoliang Dai, Peizhao Zhang, Hongbo Zhang

The approach from Meta GenAI demonstrates an efficient post-training quantization strategy for transformer-only diffusion models, allowing for 8-bit activations and 4-bit weights without any optimization or retraining, resulting in near full-precision generative quality with reduced computational cost.

Key insights

• Robust activation quantization parameters for diffusion transformers can be effectively derived from a single calibration step, particularly the initial reverse diffusion step, despite dynamic range variations.
• Dispersed weight outliers in linear layers of diffusion transformers are a critical factor degrading low-bit quantization, necessitating a finer-grained approach like group-wise quantization.
• Achieving high-quality low-bit quantization (e.g., 8-bit activations, 4-bit weights) in transformer-only diffusion models is possible without extensive retraining or optimization by addressing these specific challenges.

Summary by alphaXiv.