Reproduce “Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ryan Synk, Monte Hoover, John Kirchenbauer, Neel Jain, Alex Stein, Manli Shu, Josue Melendez Sanchez, Ramani Duraiswami and 1 more
The 'Top-k Attention' method enables transformer language models to process contexts up to 1 million tokens on commodity GPUs by offloading the full KV cache to CPU memory and employing approximate k-nearest neighbor search to retrieve only the most relevant k key-value pairs for attention. This approach reduces GPU memory usage to O(k) during decoding, maintaining over 95% of full attention performance with minimal token sampling.
Key insights
- Large language models exhibit significant sparsity in their attention mechanisms, meaning only a small fraction of input tokens contribute substantially to the attention calculation for the next token.
- Exploiting this attention sparsity allows for a substantial reduction in GPU memory and compute costs during decoding without significantly degrading model performance.
- An approximate kNN search on a CPU-resident KV cache can effectively identify these critical 'k' tokens, enabling sublinear decoding runtime and unprecedented context lengths on standard hardware.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.