Reproduce “CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huang Cheng
CacheRoute, a routing mechanism for large-scale LLM serving, concurrently plans prefix-affinity and expected server load, demonstrating a 2.3x capacity improvement over strong baselines and a 93.2% served KV-cache hit rate for a 70B LLM.
Key insights
- Achieving high LLM serving capacity at saturation requires both strong prefix affinity to maximize KV-cache hits and effective load-balanced placement to distribute the recovered work.
- A planned, periodic approach to key-to-destination assignment can resolve the cache-locality vs. load-balancing trade-off more effectively than instantaneous reactive policies.
- The efficacy of prefix-affinity routing strongly depends on the workload's inherent prefix reuse, with limited benefits for workloads with low reuse.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.