Reproduce “Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jevin Jiang, Ying Chen, Blake A. Hechtman, Fenghui Zhang, Yarong Mu
Google researchers developed Ragged Paged Attention (RPA), a high-performance attention kernel for Large Language Model (LLM) inference on TPUs, specifically designed to handle dynamic and ragged serving workloads. The kernel achieves up to 86% Memory Bandwidth Utilization in decode and 73% Model FLOPs Utilization in prefill, leading to a 2x-5x increase in token throughput when integrated into production frameworks.
Key insights
- Leveraging low-level TPU control (Pallas/Mosaic) is crucial for overcoming XLA's limitations with dynamic workloads and optimizing for TPU's unique architecture.
- Fusing KV cache updates directly into the attention kernel's compute loop effectively masks memory write latency, especially for compute-intensive prefill operations.
- Careful memory layout management, including packing dimensions and KV tensor merging, is vital for maximizing memory bandwidth utilization and minimizing padding overheads on TPUs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.