Reproduce “On Evaluating User Simulators for Interactive Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hengguang Zhou, I-Hung Hsu, Li Chen, Priyanshi Shrivastava, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

The authors evaluate metrics for simulated users by testing whether scores detect controlled replay defects. They compare altered replays with controls from the same source session and agent assignment, using checks for agent rankings, population behavior, and fidelity to each user. Their suite detects all 10 designed perturbations, versus 5–6 for prior suites, under the tested protocols.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.