Reproduce “LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lianwei Yang, Haokun Lin, Tianchen Zhao, Yichen Wu, Hongyu Zhu, Ruiqi Xie, Zhenan Sun, Yu Wang and 1 more
LRQ-DiT, developed by researchers from CASIA/UCAS and Tsinghua University, introduces a post-training quantization framework that quantizes Diffusion Transformer models to extreme low-bit settings (e.g., 3-bit weights, 4-bit activations) for image and video generation. It achieves superior performance and up to 2.45x speedup compared to FP16, enabling efficient deployment of large generative models.
Key insights
- DiT model weights require non-uniform quantization to efficiently capture their long-tailed, Gaussian-like distributions, which a logarithmic scale can effectively transform for better bit allocation.
- Activation outliers in DiTs are diverse (mild vs. salient) and necessitate an adaptive, fluctuation-aware mitigation strategy rather than uniform rotation application across all layers.
- Orthogonal rotations can suppress the L-infinity norm of activations, making them more amenable to subsequent quantization in the logarithmic domain, thereby reducing mean squared error.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.