Reproduce “Asymptotic Optimality of $cμ$-Rule for LLM Inference Scheduling under Memory Constraint” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhenghao Li, Shengbo Wang, Nian Si

The authors propose a scheduler for large language model inference that prioritizes requests by holding-cost rate times effective service rate. The rate accounts for prompt computation, key-value cache I/O, step overhead, and available memory. On a time-compressed To-B trace, it achieved 9.472 seconds average latency versus vLLM’s 35.492, with similar output-token throughput; optimality is proved only for a fluid model.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.