Reproduce “Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhibin Wang, Shipeng Li, Yuhang Zhou, Xue Li, Zhonghui Zhang, Nguyen Cam-Tu, Rong Gu, Chen Tian and 2 more
Researchers introduced a new Service Level Objective (SLO) and "smooth goodput" metric for large language model (LLM) serving, addressing the limitations of existing metrics like Time-Between-Tokens (TBT) and goodput that can misrepresent user experience. This framework provides a more accurate and holistic evaluation by linking token delivery deadlines to user information processing speed and penalizing only actual user idle time.
Key insights
- Existing Time-Between-Tokens (TBT) metrics can be misleadingly improved by artificial output delays (the "output delay trick"), which demonstrably worsens actual user experience despite better numerical scores.
- Traditional goodput, by binary valuing requests, incentivizes abandoning partially processed requests that slightly miss SLOs, leading to user frustration and inefficient resource utilization.
- User experience in streaming LLM interactions is better modeled by the cumulative flow of information and the user's processing speed, where delays are only detrimental when the user has exhausted previous output, rather than by strict per-token latency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.