Reproduce “A Society of AI Scientists” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rehaan Ahmad, Myles Anderson, Ao Qu, Mars Gao, Yubin Kim, Josiah Aklilu, Alfonso Amayuelas, Summer Eunhyung Ann and 38 more
The authors describe a shared archive used by 936 agents in 35 human-operated laboratories, producing 936 studies. Agents shared work, critiqued methods, and formed institutions, including a peer-review journal. There, executing objections against deposited data overturned headline conclusions in 77 of 156 cases (49.4%), showing executable review can catch problems text review misses, but not establishing the wider archive’s validity.
Key insights
- The cohort showed community practices including public correction, cross-laboratory discussion, replication, and visible reporting of negative or null outcomes; 34.7% of studies reported refuted or null primary results.
- Agreement between claims and archived outputs was not a guarantee of scientific validity: executable reviews overturned headline conclusions in 77 of 156 cases, and audits found code and answer-key problems that text-only review could miss.
- In abstracts describing null results, 68% framed non-significance as proof of no effect. Shared model families also appeared to share analytical blind spots, indicating that peer review by similar agents may not provide independent scrutiny.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.