Reproduce “WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More” on OpenResearch

About this paper

Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, Liqiang Nie

WKVQuant introduces a post-training quantization framework that quantizes only the weights and Key/Value (KV) cache to 4-bit precision for Large Language Models. This approach achieves memory savings comparable to full weight-activation quantization while maintaining accuracy close to weight-only methods, enabling more efficient and accurate LLM deployment.

Key insights

• Detailed memory analysis revealed that temporary activations are highly sensitive to quantization, consume less memory, and their quantization offers minimal inference speedup, making them a poor target for low-bit precision.
• A selective approach of quantizing only weights and the Key/Value (KV) cache provides a superior balance, achieving memory savings comparable to full weight-activation quantization while maintaining accuracy by avoiding problematic temporary activation quantization.
• Preserving the current token's Key and Value in full precision during auto-regressive decoding (Past Only Quantization) is critical for maintaining attention mechanism accuracy.

Summary by alphaXiv.