Reproduce “SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huzama Ahmad, Se-Young Yun
SpotAttention introduces a plug-in block-sparse routing mechanism for pretrained long-context transformers, utilizing a lightweight, learned selector trained via knowledge distillation to reduce inference costs. This method achieves up to 3.9 times faster decode throughput and supports contexts up to 512K tokens without accuracy loss, outperforming FlashAttention and other baselines.
Key insights
- A small, independently trained selector can be effectively plugged into frozen pretrained transformers to enable efficient sparse attention without requiring backbone retraining.
- Attention budget requirements are asymmetric across layers, with early and middle layers needing more context, while deeper layers utilize much sparser attention.
- Quantizing the selector's internal K-cache to ultra-low precision (INT4/FP4) significantly reduces memory footprint with negligible accuracy loss.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.