Reproduce “Fast Inference from Transformers via Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yaniv Leviathan, Matan Kalman, Yossi Matias
Speculative decoding accelerates autoregressive generation by having a smaller draft model propose tokens and a target model verify them in parallel, using probabilistic acceptance and correction to preserve the target distribution. In English-to-German translation, an 11B T5-XXL target and 77M T5-small draft achieved a 2.6× wall-time speedup in one TPU-v4 setup; extra compute is required.
Key insights
- Speculative execution, a technique from computer architecture, can be effectively generalized to stochastic language model generation to enable parallel verification of speculative token guesses.
- Even small or simple approximation models (M_q), including n-gram models, can provide meaningful acceleration by achieving non-trivial token acceptance rates.
- The method increases concurrency and can reduce memory bandwidth requirements for fetching weights and KV cache, addressing common bottlenecks in large model inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.