Reproduce “Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nils Dycke, Iryna Gurevych
The authors test whether fully automatic review generators detect when evidence no longer supports a paper’s conclusions and finding. They compare reviews of 133 accepted empirical AI/NLP papers with logic-breaking edits against surface-only controls across three measures. None showed a significant difference; this controlled result does not establish performance on flaws in real submissions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.