Reproduce “Robust KV Cache Management for LLM Serving under Output Token Length Uncertainty” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiaming Cheng, Duong The Do, Duong Tung Nguyen

Researchers from Arizona State University developed a unified optimization framework for robust KV cache management in LLM serving, leveraging Distributionally Robust Optimization to jointly determine GPU parallelism, per-class KV cache reservations, and request routing. This approach reduced the combined cost of preemption and wasted memory by up to 56% compared to heuristic baselines, demonstrating increased robustness against output length uncertainty and workload distribution shifts.

Key insights

  • The optimal KV cache reservation quantity follows a critical fractile structure, `r_i^* = L_i^in + F_i^-1(ρ/(ρ+1))`, allowing the system to automatically adapt reservations based on the relative costs of preemption and waste.
  • Integrating distributional robustness (DRO) provides resilience against evolving workload characteristics, significantly outperforming Sample Average Approximation (SAA) under distribution shifts by reducing SLO violations and tail latency.
  • Intelligent request routing across heterogeneous serving groups is crucial for maintaining feasibility and preventing queue saturation, while prefix caching further reduces queuing delays.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.