Reproduce “Efficient Multi-round LLM Inference over Disaggregated Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhao He, Youhe Jiang, Penghao Zhao, Quanqing Xu, Eiko Yoneki, Bin Cui, Fangcheng Fu
A disaggregated serving framework, AMPD, is introduced for efficient multi-round Large Language Model inference, achieving up to 967.54% better SLO attainment than existing disaggregated systems by adaptively routing prefill tasks and optimizing deployment configurations.
Key insights
- Adaptive routing, which offloads prefill tasks from prefill workers to decode workers when decode resources have slack, is crucial for balancing workload and mitigating contention in multi-round disaggregated serving.
- Systematic, workload-aware offline planning for resource allocation and parallelism for both prefill and decode phases yields empirically reliable optimal deployment configurations for complex multi-round workloads.
- Even with minimal runtime overhead, dynamically reordering tasks within prefill queues based on SLO targets significantly improves tail latencies and overall SLO attainment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.