Reproduce “SpecTr: Fast Speculative Decoding via Optimal Transport” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, Felix Yu
Researchers at Google Research developed SpecTr, a speculative decoding method that employs optimal transport theory to efficiently select from multiple candidate drafts, achieving a 2.13X wall clock speedup for large language model inference over baseline methods while preserving output quality. This approach enhances existing speculative decoding by enabling parallel processing of multiple draft sequences, leading to a 1.37X speedup compared to single-draft methods.
Key insights
- Framing multi-candidate draft selection within the Optimal Transport with Membership Cost (OTM) theory provides a rigorous, information-theoretic foundation for speculative decoding.
- The K-SEQ algorithm offers a practical and computationally efficient approximation for OTM, achieving a (1-1/e)-optimality guarantee for draft acceptance.
- Leveraging batch parallelization of draft sequences through the OTM/K-SEQ approach provides substantial inference speedups without compromising the quality or statistical distribution of the large model's output.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.