Reproduce “ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor” on OpenResearch

About this paper

Seungbeom Choi, Jeonghoe Goo, Eunjoo Jeon, Mingyu Yang, Minsung Jang

Researchers at Samsung SDS developed ELIS (Efficient LLM Iterative Scheduling System), a cloud-native system integrating an Iterative Shortest Remaining Time First (ISRTF) scheduler with a BGE-based response length predictor for Large Language Models. This system consistently reduced the average Job Completion Time (JCT) by up to 21.40% compared to FCFS scheduling by dynamically prioritizing requests based on refined predictions of remaining inference time, validated with real-world LLM service trace data.

Key insights

• Iterative prediction, where partially generated output is fed back to the predictor, significantly enhances the accuracy and reliability of estimating remaining response lengths for LLM inference.
• Real-world LLM service request arrival intervals follow a Gamma distribution (α=0.73, β=10.41), rather than a Poisson process, providing a more accurate model for simulating and evaluating LLM serving systems.
• Implementing an ISRTF scheduler with accurate, iteratively refined predictions of remaining time can substantially reduce average Job Completion Time and queuing delays for LLM inference with minimal overhead.

Summary by alphaXiv.