Reproduce “VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs” on OpenResearch

About this paper

Shahrzad Esmat, Dhawal Shah, Ali Jannesari

VarRate introduces a training-free method for compressing the KV cache in long-context Large Language Models by adaptively allocating a variable low-rank budget to each token based on its salience. This approach maintains accuracy within 0.8 points of uncompressed models on LongBench and exhibits graceful degradation (3.53–5.50 points loss) under query-agnostic reuse, significantly outperforming token-selection methods which lose 11–15 points.

Key insights

• Adaptive, salience-guided rank allocation, instead of uniform rank or token eviction, provides superior compression by ensuring critical tokens receive higher fidelity while preventing the complete loss of any contextual information.
• Retaining all tokens, even at a lower fidelity (coarse coding), offers robustness against stale importance signals and query-agnostic reuse scenarios, significantly mitigating the accuracy collapse seen in token-selection methods.
• The residual codec with anchor tokens and a shared, training-free basis effectively captures local context while minimizing computational overhead compared to re-encoding methods like KVzip.

Summary by alphaXiv.