Reproduce “Past-Future Scheduler for LLM Serving under SLA Guarantees” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruihao Gong, Shihao Bai, Siyu Wu, Yunqian Fan, Zaijun Wang, Xiuhong Li, Hailong Yang, Xianglong Liu
Researchers from Beihang University and SenseTime developed the Past-Future scheduler for LLM serving, which accurately estimates future memory requirements to maximize "goodput" (throughput under SLA guarantees). Implemented in the LightLLM framework, it consistently outperforms aggressive and conservative schedulers across various workloads and models by balancing memory utilization and preventing request evictions.
Key insights
- The distribution of LLM request output lengths remains sufficiently stable within adjacent time windows, allowing for effective dynamic prediction without explicit model-specific tuning or additional inference overhead.
- Accurately predicting the true peak memory requirement of a batch necessitates accounting for the dynamic release of memory by requests that complete earlier.
- By anticipating future memory needs, it is possible to achieve high GPU memory utilization while nearly eliminating costly request evictions, thereby maximizing goodput under SLA guarantees.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.