Reproduce “SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding” on OpenResearch

About this paper

Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, Yonatan Geifman

SPEED-Bench evaluates speculative decoding under varied prompts and serving conditions. It separates drafter accuracy testing on semantically diverse prompts from throughput measurement across fixed-length, varied-entropy workloads and inference engines. In one GPT-OSS 120B, EAGLE3, TensorRT-LLM test, random-token benchmarking overestimated output throughput by an average of 23%. This result applies to that tested configuration, not universally.

Key insights

• Drafter predictability and acceptance rates are highly sensitive to domain entropy, with low-entropy domains yielding higher average acceptance lengths (ALs).
• Synthetic data for benchmarking SD can significantly overestimate throughput and cause inaccurate latency measurements for Mixture-of-Experts (MoE) models due to skewed acceptance rates and expert imbalance.
• The optimal draft length (DL) for SD is dependent on the batch size, with longer drafts preferred at lower, memory-bound batch sizes, but shorter drafts becoming more efficient in compute-bound regimes.

Summary by alphaXiv.