Reproduce “Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rahul Thomas, Teo Kitanovski, Micah Goldblum, Arka Pal
This research introduces Dynamic Delayed Tree Expansion, a technique for multi-path speculative decoding in Large Language Models. It utilizes a neural selector to adaptively optimize tree expansion parameters, enabling existing optimal transport (OT)-based methods to achieve approximately 5% higher average throughput than the previously leading Traversal Verification across diverse models and tasks.
Key insights
- Traversal Verification consistently outperformed all other existing multi-path speculative decoding methods, including OT-based ones, in initial comprehensive benchmarks.
- OT-based methods exhibit rapidly degrading acceptance rates and increasing target-draft distribution divergence with tree depth, indicating that aggressive early branching is computationally wasteful.
- Optimizing tree expansion by delaying branching to deeper, more divergent nodes can substantially improve the block efficiency and throughput of OT-based speculative decoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.