Reproduce “Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez, Antonio Ortega, Salman Avestimehr

NOVA-KV proposes a 2-bit per-element KV cache quantization method for large language models that derives optimal non-orthogonal transforms for keys and employs a volume-equalizing vector quantization strategy. This approach maintains task accuracy and achieves up to 3.4x decode throughput improvement, particularly for long context lengths.

Key insights

  • Optimal transforms for preserving attention logits are non-orthogonal, unlike traditional MSE-optimal orthogonal transforms, requiring a specific attention-aware distortion metric.
  • A generalized Parseval relation simplifies the design of quantizers for attention-preserving transforms by converting complex, weighted distortion into standard MSE in the transform domain.
  • Volume-equalizing grouping strategies for Vector Quantization are essential to achieve near-optimal rate-distortion performance within the constraints of fixed-width memory layouts required by LLM serving engines.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.