Reproduce “FAST-Prefill: FPGA Accelerated Sparse Attention for Long Context LLM Prefill” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rakshith Jayanth, Viktor Prasanna
Researchers from USC developed FAST-Prefill, an FPGA accelerator for the prefill stage of long-context Large Language Models using dynamic sparse attention. This W8A8 precision system achieves up to 2.5x faster Time To First Token (TTFT) and 4.5x better energy efficiency compared to a GPU baseline, while maintaining comparable model accuracy.
Key insights
- Custom streaming kernels and memory-aware designs are crucial for eliminating intermediate tensor generation and irregular memory access bottlenecks in dynamic sparse attention.
- A liveness-driven caching strategy significantly reduces off-chip memory traffic and enhances data reuse for sparse, irregular KV cache access patterns on FPGAs.
- Combining DSP blocks with LUT-based bit-plane arithmetic allows FPGAs to achieve competitive matrix multiplication throughput for INT-8 operations, overcoming inherent DSP limitations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.