Reproduce “ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shihao Wang, Jiahao Chen, Yanqi Pan, Hao Huang, Yichen Hao, Xiangyu Zou, Wen Xia, Wentao Zhang and 2 more
ProphetKV introduces a user-query-driven selective recomputation framework for Retrieval-Augmented Generation (RAG) that efficiently reuses Key-Value (KV) caches. This approach recovers 96%–101% of full-prefill accuracy with only 20% recomputation, achieving up to a 5x speedup over full recomputation.
Key insights
- Existing partial recomputation methods suffer from a "crowding-out effect" by recomputing globally salient but user-query-irrelevant tokens, displacing truly essential information for a specific query.
- The utility of cross-attention in RAG is highly contingent on the user query, which can act as a "prophet" to identify critical document tokens for subsequent generation.
- Per-layer token importance can be efficiently estimated without a full prefill by only computing attention between the short user query and the precomputed, imprecise Key caches at each layer.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.