Reproduce “A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Neelesh Gupta, Peter Wang, Rajgopal Kannan, Viktor K. Prasanna
An FPGA-based accelerator for Gated DeltaNet (GDN) decode stores the recurrent state persistently on-chip to overcome memory bottlenecks. It delivers 4.5x lower latency (63 "μs" per token) and 62x greater energy efficiency (1.61 mJ per token) compared to a state-of-the-art NVIDIA H100 GPU for batch-1 inference.
Key insights
- Subquadratic sequence models like Gated DeltaNet, while reducing memory scaling, become significantly memory-bound during batch-1 decode on GPUs due to constant off-chip state transfers.
- FPGAs offer a unique advantage by providing sufficient on-chip BRAM to persistently store the entire recurrent state, converting the workload from memory-bound to compute-bound.
- Algebraic simplification of the GDN recurrence can reduce the number of required state matrix access passes, improving per-iteration efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.