Reproduce “Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving” on OpenResearch

About this paper

Haoxin Liu, Jiayi Wang, Yueshen Xu, Rui Li

Researchers at Xidian University developed a scheduling framework for chunked-prefill Large Language Model (LLM) serving that enhances fairness and latency control under heterogeneous workloads. The framework, which includes an aging-based policy and dynamic chunking, reduced mean end-to-end latency by 10.24% and improved tail latency predictability while ensuring balanced progress for diverse requests.

Key insights

• Fair scheduling in chunked-prefill LLM serving systems requires a dynamic prioritization strategy that continuously trades off between minimizing latency for short, interactive prompts and preventing starvation for long-running, compute-intensive prefill requests.
• Achieving predictable and stable latency in dynamic LLM serving environments necessitates an adaptive scheduling mechanism that estimates and controls batch execution times, rather than relying on fixed token budgets.
• Optimizing prefill efficiency and reducing scheduling overhead involves actively managing the structure of active prefills by limiting concurrency and enforcing minimum chunk sizes, thereby reducing fragmentation and promoting sustained progress.

Summary by alphaXiv.