Reproduce “AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zaifeng Pan, Michael Wang, Chris Wu, Zhengding Hu, Xinwei Qiang, Zhongkai Yu, Yufei Ding
AgentReplay makes LLM-agent serving benchmarks more repeatable by replaying recorded token-by-token interactions, dependencies, and optionally MoE expert choices and tool delays, while leaving scheduling and execution decisions to the system being tested. In evaluated tasks, replay reduced run-to-run cumulative-time variation from 25.58% to 0.66% for Astropy and from 114.93% to 1.30% for SymPy; experiments also show that matching token lengths alone can distort measured MoE workloads and execution times.
Key insights
- Greedy decoding alone does not stabilize agent workloads: repeated runs showed large variation in turns and token counts, including unproductive tool-call cycles in 31 of 50 SymPy runs.
- Token content matters beyond sequence length. Length-wise replay changed which MoE experts were heavily used and could either overstate or understate measured execution costs, depending on the configuration.
- Recorded traces can separate trajectory generation from performance testing, enabling long-horizon workloads to be replayed on smaller models; this controls the workload, but does not show that the target model could autonomously complete the task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.