Reproduce “CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhen Huang, Ruizhe Yao, Danyi Liu, Xinrui Chen, Shuwei Li, Siru Zhong, Zijian Cao, Yushan Lai and 3 more

The paper introduces CompKV, which reduces long-context KV-cache reads by selecting blocks according to estimated residual error after mean compensation, using compact statistics for attention mass and within-block logit variation. In controlled tests, residual-based ranking lowered attention KL error by 0.660% and output L2 error by 0.582%. On RULER, Qwen3-8B scored 86.0 versus 76.6 for InfLLM, while latency gains were single-layer measurements, not end-to-end speedups or guarantees over full attention.

Key insights

  • The best blocks to read exactly depend on both their attention mass and how poorly their token-level logits are represented by a block average.
  • A second-order correction to estimated block mass and the variance term improve selection quality over mass-only scoring in the reported ablations.
  • With four variance groups, estimated scores correlate 0.8604 with exact residual scores and capture 81.53% of the residual removable by exact selection in the reported experiment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.