Reproduce “RazorAttention: Efficient KV Cache Compression Through Retrieval Heads” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, Gongyi Wang

RazorAttention introduces a selective compression technique for the Key-Value (KV) cache in Large Language Models by distinguishing between retrieval and non-retrieval attention heads. This approach reduces KV cache memory usage by over 70% with less than 1% performance degradation, making LLM deployment more efficient for long contexts.

Key insights

  • Not all attention heads in LLMs function identically; some are critical "retrieval heads" for long-range dependencies, while others focus on local context.
  • Applying different KV cache strategies to specialized heads can achieve substantial memory savings without significant performance loss.
  • Aggregating information from discarded tokens into a "compensation token" can effectively preserve crucial context for non-retrieval heads.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.