Reproduce “Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sunnie S. Y. Kim, Wesley Hanwen Deng, Jennifer Wortman Vaughan, Buxin Su, Weijie Su, Alekh Agarwal, Sharon Li, Martin Jaggi and 3 more
The authors test whether rules governing reviewers’ LLM use affected peer-review outcomes at ICML 2026. Randomized assignments compared a policy barring all LLM use with one permitting limited assistance but not delegated judgment or review drafting. Acceptance rates were 27.0% and 26.5%, while permissive-policy reviews were 7% longer; results concern policy assignment, not actual LLM use.
Key insights
- Changing the assigned policy produced almost no difference in acceptance rates, paper scores, reviewer confidence, or most area-chair quality assessments.
- Permissive-policy reviews were modestly longer—roughly 199–250 additional characters, or about 7% in the paper-level analysis—without consistent evidence of objectively or independently improved quality.
- Restrictions reduced apparent LLM use but did not eliminate it: 22.5% of conservative-policy reviewers self-reported using an LLM, while 36.5% of permissive-policy users reported at least one prohibited use.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.