Reproduce “KVpop -- Key-Value Cache Compression with Predictive Online Pruning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl, David Stap, Thomas Schmied, Sebastian Böck, Günter Klambauer, Sepp Hochreiter

KVpop reduces long-context key–value cache growth by learning which older tokens to retain while protecting sink tokens and a recent window. Its scorer ranks eligible tokens for a fixed top-k cache using future-attention supervision, with an optional stateful readout at eviction. On Qwen3-4B, it reaches 0.44 Pass@1 at 88% compression, 0.97 relative to the dense teacher, across four math evaluations.

Key insights

  • Directly supervising token eviction with a 'future-attention' target provides a more robust signal for future utility, leading to more effective compression than indirect signals.
  • Delaying the eviction decision for tokens until they exit a 'protected window' allows the scorer to integrate near-future context, improving the accuracy of retention choices.
  • Efficient implementation of future-attention target computation and dynamic sparse attention kernels are crucial for practical and scalable learned KV cache compression.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.