Reproduce “AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber, Kryštof Olík, Georg Groh
The authors test whether language models can provide candidate concerns for human scientific reviewers by generating reviews of 50 ICLR submissions and 31 accepted-only Nature Communications papers and comparing concern overlap and score alignment with human reviews. GPT matched more human concerns in raw comparisons, but the gap largely narrowed after review-length normalization. These measures assess overlap and calibration, not whether concerns are correct or useful.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.