Reproduce “From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeshen Zhang, Han Zhao, Weihao Cui, Quan Chen, Yu Liu, Yongjun Deng, Jing Yang, Jiuchen Shi and 4 more
HALO improves serving of shared language models with task-specific LoRA adapters when requests have different latency targets. It overlaps base-model and adapter computation, lets requests with spare deadline slack yield compute, and buffers tokens for scheduled release. In hospital-workload tests, HALO achieved 19.5% higher throughput than QoServe under a P99 TTFT constraint; this result is limited to evaluated settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.