Reproduce “AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang and 6 more
AdaServe is an LLM serving system that leverages SLO-customized speculative decoding to efficiently handle requests with diverse latency requirements. It demonstrated up to 4.3x reduction in SLO violations and 1.9x higher goodput compared to state-of-the-art baselines.
Key insights
- The multi-SLO LLM serving problem can be formalized as a constrained optimization, where optimal token tree construction balances SLO satisfaction and expected accepted tokens.
- Decoupling the draft model's speculation (generating candidate tokens) from the selection of verified tokens significantly improves efficiency by reducing sequential decoding steps.
- Dynamic adaptation of speculative decoding parameters (depth and beam width) is essential for robustly handling fluctuating workloads and diverse SLO requirements, outperforming static strategies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.