Reproduce “An Interpretable Latency Model for Speculative Decoding in LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Linghao Kong, Megan Flynn, Michael Peng, Nir Shavit, Mark Kurtz, Alexandre Marques
An interpretable latency model is developed to characterize the end-to-end performance of speculative decoding (SD) in large language model (LLM) serving systems under dynamic, continuous batching loads. This model reveals how SD parameters like draft length and acceptance rate reshape fixed and load-dependent costs, explaining why SD speedups often diminish or reverse as server utilization increases.
Key insights
- Speculative decoding alters the effective service parameters (fixed and load-dependent costs) but does not fundamentally change the underlying load-latency relationship of continuous-batching serving systems.
- SD speedup under load involves a critical trade-off: while it reduces load-independent costs (beneficial at low load), it often increases load-dependent costs, causing speedup to diminish or reverse as server utilization rises.
- Optimal SD configurations are load-dependent; draft lengths that provide maximum speedup at batch size 1 are generally suboptimal for high-throughput serving due to increased load-dependent costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.