Reproduce “Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Samhruth Ananthanarayanan, Ayan Sengupta, Tanmoy Chakraborty
Researchers at IIT Delhi developed a physics-inspired framework to analyze Key-Value (KV) cache compression in Large Language Models, revealing that current evaluations often overlook structural issues. Their work uncovered a "hallucination safety cliff" at high compression, linked to global eviction of critical tokens, and demonstrated distinct attention routing dynamics between LLaMA and Qwen architectures.
Key insights
- All evaluated models exhibited a sharp 'hallucination safety cliff' in hallucination rates when compression approached 90%, correlating with a spike in Global Eviction Ratio (GER).
- LLaMA and Qwen models showed distinct depth-wise attention routing dynamics (inverted funnel vs. funnel-like), implying optimal compression strategies must be architecture-aware.
- Moderate compression can lead to substantial degradation in internal representations while task accuracy remains largely intact, indicating that information retention does not guarantee semantic accessibility or utilization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.