Reproduce “SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoran Qiu, Rohan Mahapatra, Rodrigo Fonseca, Mark Russinovich
SHADOWPREFILL prepares the destination model’s KV cache while another model serves an agentic call, using spare capacity for speculative prefill. This can reduce context to process after a model switch. In simulation of six GitHub Copilot coding-trace model pairs, Harvest-SLO+Host reduces switch-call latency by 14.3% and routed token cost by 32.0%; the full design is mainly evaluated in simulation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.