Reproduce “CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, Jianguo Li

A new KV cache eviction framework, CAKE, intelligently manages memory in long-context Large Language Models by adaptively allocating cache across layers based on their dynamic attention patterns and implementing a cascading prefilling strategy. This allows for a 48.63% reduction in peak memory and over 10x faster decoding while preserving or enhancing model performance on diverse tasks with substantially reduced cache sizes.

Key insights

  • Attention patterns within different LLM layers exhibit significant non-uniformity in how broadly attention is distributed (spatial dispersion) and how its focus changes over time (temporal shift), necessitating layer-specific cache allocation.
  • Optimally managing KV cache requires a 'global view' of memory allocation, where the total budget is adaptively distributed across layers based on their collective and individual attention dynamics.
  • An effective token eviction strategy must account for both the average relevance and the temporal variability of a token's attention, as its importance can fluctuate dynamically throughout the decoding process.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.