Reproduce “PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinjun Yi, Zhixin Zhao, Yitao Hu, Ke Yan, Weiwei Sun, Hao Wang, Laiping Zhao, Yuhao Zhang and 2 more
Researchers at Tianjin University and Stevens Institute of Technology developed PAT, a memory-centric attention kernel for LLM decoding that optimizes for shared prefixes across requests. It reduces redundant global memory accesses and improves GPU resource utilization through dynamic tile selection, achieving up to 21.5x speedups over FlashAttention and 17.2-68.1% lower Time Per Output Token in end-to-end LLM serving.
Key insights
- Directly addressing redundant global memory accesses for shared prefixes is critical to alleviating the memory-bound bottleneck in LLM decode attention.
- Dynamically adjusting GPU kernel tile sizes (Q-tile and KV-tile) based on CTA characteristics is essential for high hardware utilization and reducing padding/execution bubbles.
- A memory-oriented profit model for grouping shared prefixes is more effective than compute-oriented approaches for optimizing memory-bound decode operations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.