Reproduce “MoE-Spec: Expert Budgeting for Efficient Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bradley McDanel, Steven Li, Sruthikesh Surineni, Harshit Khaitan
A method named MoE-Spec from Franklin and Marshall College and Meta Reality Labs tackles the 'expert explosion' in speculative decoding for Mixture-of-Experts (MoE) large language models by intelligently managing expert activations. This approach enables 10-30% higher throughput compared to existing speculative decoding techniques while maintaining comparable output quality.
Key insights
- MoE models' expert activations across a speculative decoding draft tree are heavy-tailed, meaning a small fraction of experts account for most of the routing probability.
- The primary bottleneck for speculative decoding speedup in MoE models is the memory bandwidth consumed by loading numerous unique experts during parallel verification.
- Expert budgeting effectively decouples the depth of speculative decoding from its memory cost, allowing for greater acceleration without proportional resource overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.