Reproduce “HumanStudy-Bench: Towards AI Agent Design for Participant Simulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xuan Liu, Haoyang Shang, Zizhang Liu, Xinyan Liu, Yunze Xiao, Yiwen Tu, Haojian Jin

HumanStudy-Bench evaluates whether configured language-model agents reproduce findings from reconstructed experiments. It gives agents the original conditions, stimuli, and trial sequence, then applies the studies’ statistical analyses. Across 12 classic studies, demographics improved probabilistic agreement in 9 of 10 models, but the best agreement was partial and effect-size agreement remained low; the suite does not establish alignment for other populations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.