Reproduce “AttentionPredictor: Temporal Patterns Matter for KV Cache Compression” on OpenResearch

About this paper

Qingyue Yang, Jie Wang, Xing Li, Zhihai Wang, Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu and 3 more

Researchers from the University of Science and Technology of China and Huawei Noah's Ark Lab introduce AttentionPredictor, a learning-based method that directly forecasts next-token attention scores to compress the KV cache in Large Language Models (LLMs). This approach achieves up to 13x KV cache compression and a 5.6x decoding acceleration in long-context scenarios, maintaining less than 0.5% performance loss on diverse benchmarks.

Key insights

• Attention scores exhibit consistent, identifiable temporal patterns (re-access, sequential, seasonal) due to intrinsic LLM properties like high query self-similarity and position embeddings (e.g., RoPE).
• Directly predicting attention score distributions for critical token identification is more effective than indirect modeling or static heuristics.
• A lightweight, unified prediction model can generalize across transformer layers and attention heads, and cross-token prefetching effectively hides prediction and data transfer latencies within the per-token inference budget.

Summary by alphaXiv.