Reproduce “LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinwoo Ahn, Ingyu Seong, Akhil Kedia, Junhan Kim, Hyemi Jang, Kangwook Lee, Yongkweon Jeon
LookaheadKV learns to predict which prompt key/value states a language model will need, using learnable lookahead tokens and token-selective LoRA modules rather than generating a draft response during inference. It ranks entries for prefill-cache eviction with fewer than 0.5% additional trainable parameters. On LLaMA3.1-8B-Instruct, it scores 46.64 versus LAQ’s 43.57 at budget 64, below FullKV’s 49.88; measured eviction overhead is 14.5× lower in one tested configuration.
Key insights
- It is possible to accurately estimate future token importance for KV cache eviction by augmenting an LLM with minimal, trainable components, eliminating the need for costly explicit draft generation.
- Selectively activated low-rank adapters provide a parameter-efficient way to modify model behavior for specific tasks (like importance prediction for lookahead tokens) without altering the base model's weights.
- A ranking loss using KL divergence between predicted and actual attention-derived importance scores is effective for training the importance prediction mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.