Reproduce “Eigen Attention: Attention in Low-Rank Space for KV Cache Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, Kaushik Roy
Researchers at Purdue University developed Eigen Attention, a post-training method that compresses the Key-Value (KV) cache in Large Language Models by approximating attention operations in a lower-dimensional space. This approach achieves up to a 40% reduction in KV cache memory, enabling longer context lengths and larger batch sizes with minor performance degradation, while also showing synergistic benefits when combined with other compression techniques.
Key insights
- Key, query, and value matrices in LLMs possess an intrinsic low-rank nature, allowing them to be effectively approximated using fewer principal basis vectors.
- Compressing the embedding dimension of KV vectors is orthogonal to existing KV cache compression techniques (e.g., quantization, token pruning), enabling synergistic combinations for greater efficiency.
- Larger LLMs tend to be more resilient to KV cache compression, maintaining performance better than smaller models at equivalent compression ratios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.