Reproduce “Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ngoc Bui, Hieu Trung Nguyen, Arman Cohan, Rex Ying

Researchers from Yale University and The Chinese University of Hong Kong developed DBTrimKV, a global, learned KV cache eviction policy that predicts a token's future utility to dynamically manage memory. This approach not only substantially reduces KV memory usage but also consistently matches or surpasses the performance of full-cache inference in long-context language and vision-language tasks by mitigating attention dilution.

Key insights

  • KV eviction, when strategically applied, can enhance rather than degrade model performance in long contexts by actively suppressing distractors and mitigating attention dilution.
  • Theoretical analysis demonstrates that attention dilution occurs when irrelevant tokens scatter attention mass, and selective eviction can effectively reduce this dilution.
  • A token's future utility can be effectively learned and approximated by a query-agnostic, geometrically decaying persistence score, enabling calibrated ranking across the entire model.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.