Reproduce “DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ying Yuan, Pengfei Zuo, Bo Wang, Zhangyu Chen, Zhipeng Tan, Zhou Yu
DualMap routes requests to two prompt-derived candidate instances, choosing by cache reuse and load. It favors cached prefixes unless estimated time-to-first-token threatens the service-level objective, and migrates requests to relieve hotspots. In Tool&Agent, it reports up to a 125% increase in effective request capacity over the best baseline, counting requests meeting a 5-second TTFT limit, across two traces and eight-instance clusters.
Key insights
- The 'power of two choices' can effectively reconcile the conflicting goals of cache affinity and load balancing in distributed LLM serving, offering robust performance under varied workloads.
- Adaptive management of hash prefix length combined with dynamic, SLO-aware switching between cache-affine and load-aware routing is crucial for optimizing both cache utilization and latency.
- Selective, benefit-driven request migration can efficiently alleviate hotspots and mitigate tail latencies without significantly compromising overall cache affinity.
- Integrating consistent hashing with a dual-mapping approach allows for lightweight and efficient elastic scaling, minimizing service disruptions during cluster changes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.