Reproduce “ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhang Zhan, Lisi Chen, Shuo Shang
ResKV presents a KV cache compression technique for large language models that reconstructs the aggregated attention contributions of omitted tokens into a separate residual cache. This approach integrates these reconstructed contributions with exactly retained tokens via a shared softmax normalization, achieving consistent performance improvements on long-context benchmarks while operating within fixed memory budgets.
Key insights
- The collective attention contribution of numerous individually low-scoring tokens (the 'residual') can represent a substantial aggregate mass critical for robust attention computation.
- By explicitly representing this residual in a separate, compact cache, ResKV can restore omitted attention mass without corrupting the exact memories of high-priority tokens.
- Integrating the residual and main cache contributions through a single shared softmax normalization ensures proper re-weighting and participation in attention dynamics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.