Reproduce “On Evaluating User Simulators for Interactive Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hengguang Zhou, I-Hung Hsu, Li Chen, Priyanshi Shrivastava, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh
The authors evaluate metrics for simulated users by testing whether scores detect controlled replay defects. They compare altered replays with controls from the same source session and agent assignment, using checks for agent rankings, population behavior, and fidelity to each user. Their suite detects all 10 designed perturbations, versus 5–6 for prior suites, under the tested protocols.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.