Reproduce “Threshold-Based Exclusive Batching for LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weifang Zhang, Yuzhou Nie, Bowen Pang, Guangrui Ma, Shining Wu
The authors propose threshold-based exclusive batching for large language model inference and a hybrid scheduler that selects between exclusive and mixed batching online. It estimates conditions and compares mixed-batch costs with exclusive-batching costs. On WildChat with Qwen3-8B and an L40S, exclusive batching achieved 14.70 requests per second versus 10.36 for vLLM v1, but reversals across workloads make the advantage conditional.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.