Reproduce “Attn-QAT: 4-Bit Attention With Quantization-Aware Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peiyuan Zhang, Matthew Noto, Wenxuan Tan, Chengquan Jiang, Will Lin, Wei Zhou, Hao Zhang
A Quantization-Aware Training (QAT) method named Attn-QAT enables reliable 4-bit attention, recovering quality degradation in diffusion models and large language models to near BF16 levels while achieving up to 1.74x throughput speedups on NVIDIA Blackwell GPUs.
Key insights
- Quantization-aware training (QAT) can fully compensate for 4-bit attention errors, rendering explicit outlier mitigation heuristics largely unnecessary.
- Maintaining numerical consistency between forward and backward passes, especially for recomputed activations within fused attention kernels, is crucial for training stability.
- Dedicated high-precision intermediate outputs during the forward pass are required to preserve the algebraic identities of memory-efficient backward passes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.