Reproduce “More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression” on OpenResearch

About this paper

Aryan Sood, Tanvi Sharma, Vansh Agrawal

A new KV-cache compression framework, LASER-KV, introduces a hybrid token selection policy that combines global attention consensus with Locality Sensitive Hashing to enhance long-context semantic recall in Large Language Models. This approach achieved stable performance on complex reasoning tasks up to 128,000 tokens on the Babilong benchmark, where other state-of-the-art compression methods significantly degraded or failed.

Key insights

• Relying solely on attention scores for KV-cache compression creates "brittle memory" and often discards structurally relevant tokens, leading to performance collapse at extreme context lengths.
• A hybrid token selection strategy that combines direct attention (for precision) with Locality Sensitive Hashing (for recall of structural similarity) yields more robust and reliable long-term memory for LLMs.
• An accumulative budgeting approach with a tunable protection divisor allows for stable performance by effectively balancing the retention of recent tokens with semantically critical historical context.
• The Babilong benchmark provides a strong tool for evaluating the stability and recall capabilities of KV-cache compression methods, especially in synthetic, long-context reasoning tasks.

Summary by alphaXiv.