Reproduce “eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization” on OpenResearch

About this paper

Pei-Chun Su

A compression method for large language model KV caches, eOptShrinkQ, processes cache blocks as spiked random matrices, applying optimal spectral denoising to separate low-rank shared context from token-specific information. The full-dimensional, isotropic residual is then quantized with a standard per-vector scalar quantizer, reducing KV cache memory footprint by up to 7x while preserving or improving attention fidelity and downstream task performance.

Key insights

• KV cache blocks inherently exhibit a spiked random matrix structure, with outlier singular values representing shared contextual information and a bulk of singular values corresponding to token-specific content.
• Optimal spectral denoising effectively isolates the low-rank signal component, restoring the isotropic properties of the residual matrix, which is a key precondition for effective per-vector scalar quantization.
• Removing the anisotropic signal component through denoising inherently reduces inner product bias, enabling near-zero bias without additional bit overhead for explicit correction mechanisms.

Summary by alphaXiv.