Reproduce “ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, Minyi Guo

ClusterKV introduces a recallable KV cache compression method for large language models that groups tokens into semantic clusters, reducing memory consumption and improving inference speed. This approach achieved up to 2x latency reduction and 2.5x throughput increase for 32k contexts, demonstrating accuracy on par with full KV cache inference at a 1k-2k token budget.

Key insights

  • Tokens semantically proximate in their key vector space tend to exhibit similar attention weights, suggesting that semantic clustering can effectively capture dynamic token importance for recall.
  • Grouping tokens into semantic clusters significantly reduces 'internal fragmentation' compared to position-based page recall, allowing for more precise and efficient utilization of the limited KV cache budget.
  • Overlapping clustering computation with other Transformer operations and optimizing key components with custom CUDA kernels can render the overhead of dynamic semantic clustering negligible, enabling efficient real-time inference.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.