Reproduce “Evolving Infra: Agent-Adaptive LLM Serving at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Kang, Weili Xu, Ziyang Li, Zheng Du, Hanchen Li, Tian Xia, Song Bian, Tushar Krishna and 3 more
Evolving Infra adapts prefill–decode allocation for agent workloads while keeping KV cache state available across nodes as roles change. Its planner estimates phase workloads, while routing and eviction manage reusable cache. On a shifting Kimi K2.6 workload, mean end-to-end request latency was 38.7 seconds, versus 83.1 with Dynamo. P95 TTFT rose above Dynamo on decode-heavy Putnam at two tested concurrency levels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.