Reproduce “Adaptive Draft-Verification for Efficient Large Language Model Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xukun Liu, Bowen Lei, Ruqi Zhang, Dongkuan Xu
ADED uses adaptive trigram statistics and Monte Carlo Tree Search to propose continuations for joint target-model verification. It updates proposals using recent model outputs. On MT-Bench with Vicuna-7B, ADED reports 12.95 ms latency versus REST’s 16.31 ms, and 2.42 accepted tokens per target-model pass versus REST’s 1.97; these results measure decoding performance, not answer quality or universal speedup.
Key insights
- A dynamically updated tri-gram matrix, refined by feedback from the LLM's actual outputs, effectively approximates the LLM's evolving output distribution and improves draft quality.
- Monte Carlo Tree Search (MCTS) with PUCT scores efficiently balances exploration and exploitation in draft generation, leading to more coherent and acceptable token sequences.
- Tri-grams provide the optimal contextual depth for the adaptive LLM representative, achieving higher average acceptance rates compared to less or overly constrained n-gram lengths.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.