Reproduce “Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ales Kubicek, Andrea Caciolai, Liangzu Peng, Sukrit Sumant, Benjamin Hoover, Fraser Greenlee, Ruairidh M. Battleday, Radim Urban and 2 more
The authors introduce Latent Sparse Attention (LSA), which groups similar keys and searches group representatives before retrieving selected original keys for attention. Its Flat variant scans all group centroids, while Tree uses hierarchical search. On GLM-5.3-FLASH, Flat makes one layer’s prefill 22.7× faster at 10M tokens than the model’s sparse-attention baseline; this is not an end-to-end speedup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.