Reproduce “VeriCache: Turning Lossy KV Cache into Lossless LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiayi Yao, Samuel Shen, Kuntai Du, Shaoting Feng, Dongjoo Seo, Rui Zhang, Yuyang Huang, Yuhan Liu and 2 more
VeriCache uses a compressed KV cache to draft tokens, then verifies them against the full cache and corrects the first mismatch. This lets lossy compression propose tokens without becoming the source of truth. In long-context decoding, Llama-70B reaches 256 tokens per second versus 102 with Full KV, using KVzip at 0.2 compression; greedy decoding is the correctness guarantee.
Key insights
- Lossy KV cache compression causes 'per-token bias accumulation,' leading to an exponential decay in the likelihood of producing the exact full-KV output over long generation sequences.
- Speculative execution can effectively convert lossy KV cache compression into an accelerator for exact computation by drafting with the compressed cache and verifying against the full cache.
- Cross-resource staggering and high acceptance rates are critical for amortizing the cost of full-KV verification and achieving substantial throughput gains without compromising output fidelity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.