Reproduce “HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization” on OpenResearch

About this paper

Wenxuan Liu, Sai Qian Zhang

HQ-DiT introduces an efficient hybrid FP4 quantization framework for Diffusion Transformers, enabling both weights and activations to be compressed to 4-bit precision. This approach preserves the model's high generative quality, achieving an FID of 9.25 on conditional ImageNet 256x256 at FP4, while significantly reducing memory footprint by 2.13x and providing a theoretical 5.09x speedup.

Key insights

• Channel-level outliers in DiT activations are a primary impediment to aggressive low-bit quantization, and orthogonal transforms like Hadamard can effectively manage them.
• Fusing Hadamard transforms into weights offline significantly reduces online computational overhead, making the outlier mitigation practical for inference.
• Dynamically selecting the optimal FP exponent and mantissa bit allocation per layer, based on data distribution, is superior to using fixed FP formats for weight quantization.

Summary by alphaXiv.