Reproduce “Fast Inference via Hierarchical Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Clara Mohri, Haim Kaplan, Tal Schuster, Yishay Mansour, Amir Globerson
Researchers from Harvard University, Google Research, Tel Aviv University, and Google DeepMind introduce Hierarchical Speculative Decoding (HSD), an approach that uses multiple draft models in a hierarchy to accelerate Large Language Model inference. This method offers up to 1.17x additional speedup over single-drafter speculative decoding while mathematically guaranteeing the target model's output distribution.
Key insights
- A multi-drafter hierarchical approach can achieve greater inference speedups than single-drafter speculative decoding while preserving the target model's output distribution.
- The complex problem of optimizing a hierarchical speculative decoding system can be efficiently solved by mapping it to a Generalized Shortest Path algorithm.
- Models pre-trained with early-exit objectives, like LayerSkip, demonstrate greater speedup potential within the HSD framework compared to general models adapted for early exits.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.