Reproduce “VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu, Ziwei He, Zhouhan Lin
VQKV, a training-free vector quantization method, achieves high-fidelity and high-ratio compression for Large Language Model (LLM) Key-Value (KV) caches. It enabled LLaMA3.1-8B to reach an 82.8% compression ratio while retaining 98.6% of baseline performance on LongBench and extended the maximum generation length by 4.3 times on a single RTX 4090 GPU.
Key insights
- Vector quantization, specifically RSimVQ, can effectively achieve both high compression ratios and high reconstruction fidelity for LLM KV caches in a training-free manner.
- The residual structure of RSimVQ robustly handles complex KV cache characteristics, such as heterogeneous frequency variations introduced by Rotary Position Embeddings (RoPE).
- The value codebook is more sensitive to configuration (number and size) than the key codebook, implying its critical role in preserving long-context modeling performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.