Reproduce “Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yihao Zheng, Jingzhe Jiang, Dejiang Zhu, Zhiyuan Tan, Yang Tian, Tao Wang, Minchen Yu
A two-week trace from an agent platform operating across more than 10,000 GPUs links application behavior to inference and context-management demands across 11.69 million requests, 1.42 million tasks, and 948,400 sessions. The analysis finds highly concentrated workloads, little actual overlap among logically parallel requests, and context reuse that often crosses task boundaries; an offline retention model also shows that category-specific cache TTLs can reduce modeled recomputation at some retention budgets.
Key insights
- Workload contribution is highly uneven: the top 5% of sessions by request count produce 56.1% of requests, while a high-demand group comprising 14.3% of sessions accounts for 98.5% of measured context residency.
- Logical branching does not imply concurrent inference: only 0.225% of sibling request pairs overlap in service time, and serial tasks—45.8% of tasks—generate 77.6% of requests.
- Context continuity can cross task boundaries: requests in the same episode across tasks show 80.6% token reuse, compared with 44.7% for requests in the same task separated by an episode boundary.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.