Reproduce “Fairness in Serving Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica
Virtual Token Counter (VTC) adapts fair-queueing principles to multi-tenant LLM inference by allocating service according to weighted input and output tokens rather than request counts. Integrated with continuous batching, it provides bounded service disparity, supports weighted priorities, and maintains nearly the throughput of FCFS while reducing unfairness on synthetic and LMSYS-derived workloads.
Key insights
- Fairness for LLM serving is better expressed as equitable token-level service than as an equal number of completed requests.
- For continuously backlogged clients, VTC bounds service disparity by twice a quantity determined by maximum input size and batch capacity; the gap does not grow with the duration of overload.
- Work conservation and isolation need not be opposites: clients below their fair share can be served promptly while overloaded clients use remaining capacity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.