Reproduce “Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ngoc Bui, Hieu Trung Nguyen, Arman Cohan, Rex Ying
Researchers from Yale University and The Chinese University of Hong Kong developed DBTrimKV, a global, learned KV cache eviction policy that predicts a token's future utility to dynamically manage memory. This approach not only substantially reduces KV memory usage but also consistently matches or surpasses the performance of full-cache inference in long-context language and vision-language tasks by mitigating attention dilution.
Key insights
- KV eviction, when strategically applied, can enhance rather than degrade model performance in long contexts by actively suppressing distractors and mitigating attention dilution.
- Theoretical analysis demonstrates that attention dilution occurs when irrelevant tokens scatter attention mass, and selective eviction can effectively reduce this dilution.
- A token's future utility can be effectively learned and approximated by a query-agnostic, geometrically decaying persistence score, enabling calibrated ranking across the entire model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.