Reproduce “S2-Attention: Hardware-Aware Context Sharding Among Attention Heads” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xihui Lin, Yunan Zhang, Suyu Ge, Liliang Ren, Barun Patra, Vishrav Chaudhary, Hao Peng, Xia Song
The S2-ATTENTION kernel and Head-Heterogenous Strided Transformer (HHST) architecture enhance the practical efficiency of large language models. The approach delivers up to 2.8x faster training and 4.5x faster inference (at 256K context for 7B models) while maintaining competitive pre-training quality and achieving perfect 128K context retrieval accuracy.
Key insights
- Translating theoretical sparse attention efficiency into practical gains requires hardware-aware kernel design to optimize GPU memory access patterns, particularly I/O between HBM and SRAM.
- Achieving effective memory savings during decoding necessitates 'KV-efficient sparsity' patterns that support predictable KV eviction based on absolute positions rather than dynamic, sample-dependent schemes.
- A hybrid architectural approach, combining heterogeneous strided sparse attention with a minority of dense layers, effectively balances computational efficiency with strong model quality and long-context capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.