Reproduce “More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Noam Elata, Itay Lamprecht, Mikey Shechter, Daniel Ohayon, Itay Hubara, Daniel Soudry
SAGA separates key and value head counts, using fewer key heads to reduce query–key work while retaining value capacity for sparse attention. Paired with Atop-N, an approximate selector, it achieves up to 2.6× end-to-end per-token decoding speedup on Llama 3.2-1B at 128K context and batch size 16 versus full-attention GQA. SAGA quality evaluation reaches models up to 1.5B parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.