Reproduce “Evolving Infra: Agent-Adaptive LLM Serving at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Kang, Weili Xu, Ziyang Li, Zheng Du, Hanchen Li, Tian Xia, Song Bian, Tushar Krishna and 3 more
Evolving Infra separates node phase allocation from shared KV-cache management, allowing nodes to switch between prefill and decode while cached context remains accessible. An online planner adapts the split, and a branch-aware directory guides cache eviction. On tested workloads, it reduced mean request latency, though Putnam P95 TTFT sometimes exceeded Dynamo’s.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.