Reproduce “Hydragen: High-Throughput LLM Inference with Shared Prefixes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini
Hydragen significantly accelerates Large Language Model inference by redesigning the attention mechanism to efficiently process batches with shared prefixes, transforming memory-bound operations into hardware-friendly, compute-bound ones. This approach delivers up to 32x higher throughput compared to state-of-the-art baselines in shared prefix scenarios, particularly benefiting long contexts and large batch sizes.
Key insights
- The global softmax operation in attention can be managed through a denominator rescaling trick, enabling precise decomposition of attention across subsequences (prefix and suffix).
- By batching attention over identical shared prefixes across multiple sequences (inter-sequence batching), redundant memory reads of the prefix KV cache are drastically reduced, and compute utilization is improved.
- Transforming memory-bound matrix-vector products into compute-bound matrix-matrix products allows for greater utilization of specialized hardware like GPU tensor cores, leading to significant throughput gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.