Reproduce “PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun

PremiseCheck checks factual errors in peer-review comments by having an LLM decompose each comment into premises without reading the paper, then routing and verifying those premises with a decision model. It scores each comment by its weakest premise. On NLP comments, no-reasoning PremiseCheck reaches 63.1 AUROC versus 63.2 for holistic GPT-5.5, at 24.6% of the cost and 45.0% of the latency; non-inferiority was established only for NLP.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.