Reproduce “Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dan Peng, Zhihui Fu, Zewen Ye, Zhuoran Song, Jun Wang
SharePrefill introduces a training-free sparse attention method for Large Language Models that reduces the computational cost of long-context prefilling by dynamically sharing attention patterns among similar heads. This approach achieves approximately 1.78x speedup over FlashAttention at 128K context length, maintaining accuracy comparable to full attention baselines on benchmarks like InfiniteBench.
Key insights
- Attention score maps within the same layer of LLMs exhibit strong and exploitable similarity between different attention heads.
- The observed inter-head similarity patterns remain consistently stable across a wide range of diverse inputs and tasks.
- These empirical findings enable the reliable clustering of attention heads, forming the basis for an efficient pattern sharing mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.