Reproduce “AI Reviews Are Biased Towards the Appendix” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Saptarshi Ghosh, Mohammad Saim, Tianyu Jiang

The authors trace weaknesses in AI-generated reviews of research papers to the sections supporting them. They define a weakness as appendix-related when any supporting evidence comes from an appendix, then compare model and human reviews. Recent GPT and Claude models draw 62.00% of their weaknesses from appendices, versus 8.40% in human reviews from ICLR 2021–24; the samples differ in years and papers, and the study does not assess criticism accuracy or usefulness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.