Reproduce “BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang and 6 more
BurstGPT introduces a large-scale, real-world workload dataset for LLM serving systems, collected from Azure OpenAI GPT services over 213 days, encompassing 10.31 million traces. It provides a realistic benchmark for evaluating and optimizing LLM serving systems, revealing specific performance, stability, and reliability challenges under dynamic, real-world conditions.
Key insights
- Real-world LLM workloads exhibit distinct and often unstable burstiness in user request concurrency, posing greater challenges to serving systems compared to uniform or non-LLM workloads.
- User conversation patterns are highly varied, with a significant portion consisting of single requests and complex inter-request interval distributions.
- LLM serving systems experience notable response failure rates, particularly in conversational services, often attributed to memory bottlenecks like KV cache management under bursty conditions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.