Reproduce “ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seungwoo Jeong, Moohyun Song, Juhyun Park, Kyungyong Lee

Researchers at Hanyang University developed ShuntServe, a system that optimizes large language model (LLM) serving on heterogeneous spot GPU clusters to reduce operational costs and enhance reliability. The system achieves up to 1.17x higher throughput and over 30% greater cost efficiency compared to baselines, while minimizing downtime during spot instance interruptions.

Key insights

  • Accurate performance estimation for LLM serving on heterogeneous hardware can be achieved with a roofline model-based analytical approach requiring only one-time calibration, reducing profiling overhead.
  • Jointly optimizing model placement, parallelization strategies, and layer assignments for heterogeneous GPU clusters, considering distinct prefill and decode phase bottlenecks, significantly improves overall throughput and cost efficiency.
  • Decoupling model weight and KV cache management from the inference engine via a shared tensor store enables concurrent pipeline initialization, effectively mitigating downtime caused by spot instance interruptions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.