Reproduce “HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, Huan Li

HARD-KV presents a unified framework that reconciles the algorithmic benefits of dynamic, head-adaptive Key-Value (KV) cache compression with the static memory requirements of high-performance LLM inference engines. The system achieves up to 2x throughput improvements and enhanced accuracy for 10k+ token generation in mathematical reasoning tasks by employing a cascade cache hierarchy, logits calibration, and index regularization.

Key insights

  • Head-adaptive KV compression can substantially enhance accuracy in complex, long-context reasoning tasks by dynamically retaining critical tokens.
  • Reconciling the dynamic memory requirements of adaptive compression with the static constraints of high-performance inference engines is essential for practical efficiency and deployment.
  • Logits calibration provides a unified probabilistic budgeting mechanism, allowing diverse KV selection methods to be consistently controlled and compared, thereby improving sparsity utilization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.