Reproduce “MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sheng Qiang, Ruiwei Chen, Yinpeng Wu, Jinyu Gu, Zhichao Hua, Yubin Xia, Binyu Zang, Haibo Chen

MosaicKV introduces a dynamic two-dimensional KV cache compression framework, enabling large language models to serve extremely long contexts more efficiently. It reduces KV cache memory by approximately 3 times while achieving up to 7.3 times higher throughput and a minimal 1.76% average accuracy loss across evaluated models.

Key insights

  • Element importance within the KV cache is non-uniform across individual vectors and varies across different cache regions, indicating the need for dynamic, adaptive compression strategies.
  • The decode stage saturates GPU memory bandwidth but leaves GPU CUDA cores and CPU underutilized, presenting an opportunity for offloading and custom kernel development for sparse data.
  • Aggressive two-dimensional compression is feasible without significant accuracy loss if important KV elements are intelligently identified and preserved, and if compression strategies are dynamically adapted.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.