Reproduce “FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aniruddha Nrusimha, William Brandon, Mayank Mishra, Yikang Shen, Rameswar Panda, Jonathan Ragan-Kelley, Yoon Kim
FlashFormer targets low-batch text decoding by fusing a transformer forward pass into one GPU kernel and overlapping data movement for later work with current computation. For Llama 3.1 8B in bfloat16, it reports 182 decoded tokens per second versus GPTFast’s 159 at batch size 1 on one H100. This is decoding-throughput evidence, not full-request latency or broad hardware validation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.