Reproduce “Preble: Efficient Distributed Prompt Scheduling for LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, Yiying Zhang
Preble coordinates request placement across data-parallel GPUs to reuse shared prompt prefixes without overloading GPUs. It combines cache-aware routing with prompt-aware load estimates and local policies. Against round-robin SGLang with prefix caching, it improves average end-to-end request latency by 1.5×–14.5× and p99 latency by 2×–10× across evaluated workloads and setups. Gains are smaller for programming workloads, where decoding dominates more.
Key insights
- Prompts are significantly longer than outputs in contemporary LLM workloads, making prefill optimization critical.
- Prompt sharing is prevalent (85% to 97% of tokens shared) across diverse LLM use cases, highlighting the potential for KV state reuse.
- Real-world LLM request arrival patterns and prompt sharing structures are highly dynamic and varied, necessitating adaptive scheduling solutions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.