Reproduce “PaperCritic: Training AI Reviewers to Assess the Soundness of Papers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyun Ryu, Ji Yong Cho, Eunho Yang, Moontae Lee, June Yong Yang, Sean Welleck, Carolyn Rose

PaperCritic trains an assistant to flag soundness-related concerns about papers as review points. It uses labeled reference concerns for supervised fine-tuning and reinforcement learning, rewarding point soundness, substantiveness, and coverage. On PaperSound-23K, PaperCritic-4B achieves macro precision 47.4, recall 51.4, and F1 45.1, versus 23.7 for the strongest conventional trained 4B baseline; it addresses soundness, not full peer review.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.