Reproduce “Residual vector quantization for KV cache compression in large language model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ankur Kumar
Applying Residual Vector Quantization (RVQ) to compress the KV cache in large language models enables a 5.5x memory reduction compared to half-precision storage. This approach largely preserves model performance across various benchmarks, making LLM deployment more feasible on memory-constrained devices.
Key insights
- Increasing RVQ's residual depth (K) consistently improves model performance, recovering most unquantized model performance at K=8 with less than 1% average drop on benchmarks.
- Non-contiguous channel grouping generally outperforms contiguous grouping, especially for compressing the key matrix, suggesting a benefit from distributed information.
- Fine-tuning attention weights concurrently with codebook learning yields consistent performance improvements of 0.5-1.5% across tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.