Reproduce “Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jang-Hyun Kim, Dongyoon Han, Sangdoo Yun

Fast KVzip compresses an LLM’s key-value cache for long-context answering by training small gates to predict which earlier-token entries to keep, avoiding KVzip’s context-reconstruction step at inference. On SCBench-KV with Qwen2.5-7B-1M, at a 30% cache budget, it reports 68.2 accuracy in 13.1 seconds, versus KVzip’s 65.0 in 65.4 seconds; this is one tested setting.

Key insights

  • Reconstruction-based importance targets can guide gates toward retaining context useful across different tasks, rather than only information relevant to a particular query.
  • The learned gates add little inference overhead: batched decisions during decoding keep reported gating cost near 1% of model forward-pass latency.
  • Retention patterns differ across layers and heads; some heads sparsely preserve structural tokens such as punctuation and line breaks, while others retain more frequent tokens.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.