Reproduce “HumanStudy-Bench: Towards AI Agent Design for Participant Simulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuan Liu, Haoyang Shang, Zizhang Liu, Xinyan Liu, Yunze Xiao, Yiwen Tu, Haojian Jin
HumanStudy-Bench evaluates whether configured language-model agents reproduce findings from reconstructed experiments. It gives agents the original conditions, stimuli, and trial sequence, then applies the studies’ statistical analyses. Across 12 classic studies, demographics improved probabilistic agreement in 9 of 10 models, but the best agreement was partial and effect-size agreement remained low; the suite does not establish alignment for other populations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.