Reproduce “AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models” on OpenResearch

About this paper

Zunhai Su, Wang Shen, Linge Li, Zhe Chen, Hanyu Wei, Huangqi Yu, Kehong Yuan

Researchers from Tsinghua University and Huawei Technologies Co., Ltd. introduce AKVQ-VL, an attention-aware and outlier-free 2-bit quantization method specifically for the Key-Value cache in Vision-Language Models (VLMs). The approach achieves nearly lossless 2-bit quantization, leading to a 2.13x reduction in peak memory usage and a 2.46x increase in inference throughput, enabling more efficient deployment of VLMs.

Key insights

• Vision-Language Models (VLMs) exhibit unique attention patterns (Text-Salient Attention, Pivot-Token-Salient Attention) that differ from Large Language Models (LLMs), requiring VLM-specific KV cache quantization.
• Outliers in KV tensors, particularly in the Key dimension, significantly impede effective low-bit quantization (e.g., 2-bit) for VLMs.
• The Walsh-Hadamard Transform (WHT) can effectively mitigate outliers in KV tensors, making them more amenable to aggressive low-bit quantization without substantial accuracy loss.

Summary by alphaXiv.