Reproduce “AdaSplash-2: Faster Differentiable Sparse Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nuno Gonçalves, Hugo Pitorro, Vlad Niculae, Edoardo Ponti, Lei Li, Andre Martins, Marcos Treviso
ADASPLASH-2 introduces an optimized GPU implementation for α-entmax attention, utilizing a novel histogram-based initialization and a four-phase kernel to accelerate normalizer computation. This method achieves training throughput competitive with FlashAttention-2, particularly in sparse regimes, and enhances language models' long-context generalization and in-context learning capabilities.
Key insights
- Approximating attention scores with an on-chip histogram can significantly accelerate the convergence of the α-entmax normalizer calculation, enabling faster training.
- Efficiently exploiting the dynamic sparsity inherent in α-entmax attention with hardware-aware GPU kernels allows it to achieve per-step training times comparable to or faster than highly optimized dense attention (e.g., FlashAttention-2) in sparse scenarios.
- Principled, differentiable sparse attention, when made computationally efficient, contributes to improved long-context generalization and enhanced in-context learning in transformer language models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.