Reproduce “Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse
CASCADE coordinates request scheduling and multi-tier KV-cache management in LLM serving by dynamically tracking a "per-request latency budget." This approach achieves up to 2.4x the SLO-satisfied goodput compared to FCFS baselines and reduces SLO violations by 40% on production workloads, maintaining fairness across diverse request lengths.
Key insights
- Explicitly quantifying and dynamically tracking a 'per-request latency budget' provides a unified metric for coordinated decision-making across request scheduling and multi-tier KV-cache management.
- Prioritizing requests with the least remaining budget (least-remaining-budget-first) in the primary queue effectively reduces head-of-line blocking without penalizing long-context requests that still have ample headroom.
- Conditional deep-tier KV-cache restoration, executed only when its transfer latency fits within a request's remaining budget, prevents 'cache-hit SLO violations' and optimizes memory hierarchy utilization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.