Reproduce “HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ting Sun, Penghan Wang, Fan Lai
HyGen introduces an elastic serving system that co-locates latency-sensitive online and throughput-oriented offline LLM workloads on a single inference instance. The system achieves substantial throughput gains and improved resource utilization by opportunistically processing offline requests, all while strictly preserving online request latency Service-Level Objectives.
Key insights
- Accurate, lightweight latency prediction and dynamic SLO-aware profiling are essential for effectively balancing strict online latency guarantees with opportunistic offline processing.
- Leveraging prefix sharing among offline requests significantly boosts throughput by enabling efficient KV cache reuse and reducing redundant computation.
- An elastic, interference-aware scheduling approach can successfully co-locate distinct LLM workload types on shared hardware without violating latency-critical performance targets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.