Reproduce “BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures” on OpenResearch

About this paper

Xiannan Hu, Tianyou Zeng, Xiaoming Yuan, Liwei Song, Guangyuan Zhang, Bangzheng He

BestServe introduces a three-level hierarchical framework to efficiently select optimal serving strategies for Large Language Models, determining configurations that maximize goodput for various architectures within minutes on a single CPU. This framework achieves prediction accuracies with average absolute relative errors of 8.6% to 12.1% across typical inference scenarios, significantly reducing the need for time-consuming and resource-intensive empirical benchmarking.

Key insights

• CPU-GPU dispatch latency is a critical, often overlooked, bottleneck in the decode phase of LLM inference, causing it to be dispatch-bound rather than memory-bound.
• Optimal LLM serving strategies are highly scenario-dependent, with neither collocation nor disaggregation architectures consistently outperforming the other across all operating conditions.
• A lightweight and accurate predictive framework can effectively replace extensive and costly empirical benchmarking for optimizing LLM deployment strategies.

Summary by alphaXiv.