Reproduce “AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifeng Gu, Zicong Jiang, Jianxiu Jin, Kailing Guo, Ziyang Zhang, Xiangmin Xu
AhaKV is an adaptive KV cache eviction strategy that addresses positional bias in attention scores and integrates value vector information, achieving state-of-the-art accuracy on long-context benchmarks and short-text tasks while reducing KV cache memory by over 90% and improving inference speed.
Key insights
- Accumulated attention scores in causal transformers are inherently positionally biased, leading to a disproportionate eviction of later tokens.
- Standard softmax attention distributions flatten as sequence length increases, diminishing the ability to differentiate truly important tokens.
- Value vectors contain distinct contextual information that can be effectively leveraged to provide a more holistic and accurate measure of token importance for eviction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.