Reproduce “Behavior-Preserving KV Cache Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Doo Hwan Hwang, Junyoung Jang, Junho Na, Hosung Lim, Kee-Eung Kim
The method compresses language-model KV-cache entries by estimating how each eviction changes the next-token distribution, using pre-eviction statistics and a residual-path approximation rather than candidate-specific masked forward passes. It ranks entries by estimated KL divergence. On RULER 8K, LLaMA3.1-8B scores 61.2 with 90% evicted versus CAOTE's 51.0; the full-cache score is 80.2, in this evaluated prefill-time setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.