Reproduce “Quantize What Counts: More for Keys, Less for Values” on OpenResearch

About this paper

Mohsen Hariri, Alan Luo, Weicong Chen, Shaochen Zhong, Tianyi Zhang, Qifan Wang, Xia Hu, Xiaotian Han and 1 more

Researchers from Case Western Reserve University, Rice University, and Meta developed a geometry-driven principle for mixed-precision KV cache quantization, demonstrating that allocating higher precision to keys than values rigorously reduces quantization error and preserves model accuracy. This approach enables substantial memory savings, improving LLM inference efficiency across diverse models and tasks.

Key insights

• Key projection matrices and their resulting activations consistently exhibit larger spectral and Frobenius norms and higher information density compared to value components, rendering them more sensitive to quantization errors.
• For any given memory budget, prioritizing higher precision for keys over values rigorously minimizes overall quantization error, leading to better preservation of model accuracy.
• The proposed geometry-driven 'More for Keys, Less for Values' principle is orthogonal and synergistic with existing KV cache optimization techniques like outlier redistribution, enabling cumulative performance gains.

Summary by alphaXiv.