Reproduce “Language-model groups overstate consensus when replaying human deliberation on a reasoning task” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tengfei Shao
The authors replay human deliberation on a Wason reasoning task by seeding language-model agents with participants’ pre-discussion answers. The agents deliberate without the answer key or discussion. In participation-matched groups, full consensus was 51.1% for humans, 85.2% for chat agents, and 95.6% for reasoning-mode agents. On a reparameterized task, 74.0% of reasoning-mode groups agreed on a wrong answer, showing that agreement does not establish accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.