Reproduce “Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, Mosharaf Chowdhury
Andes, an LLM serving system from the University of Michigan and UIUC, defines and optimizes Quality-of-Experience (QoE) for text streaming services by employing a token-level preemptive scheduler. It achieves a 97% request success rate at 0.95 QoE or higher and reduces Time-to-First-Token by 8.7 seconds by dynamically managing GPU resources.
Key insights
- A formal user-centric QoE definition enables more effective optimization of LLM text streaming, addressing initial delays, sustained slow delivery, and intermittent pauses.
- Token-level preemptive scheduling, when intelligently designed with overhead awareness, can significantly improve QoE by prioritizing prompt delivery and smooth token flow while mitigating head-of-line blocking.
- Aligning token delivery with user consumption speed, facilitated by client-side pacing, avoids wasted computation from over-generation and allows for more efficient GPU resource allocation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.