Reproduce “ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kihyun Na, Injung Kim
ReviewMap separates peer-review comments into assessment (ASM), which evaluates a paper’s claims or execution, and calibration (CAL), which judges contribution value, then codes attributes describing targets and textual grounding. Across 17,654 comments from 500 English ICLR 2026 papers, models met the study’s act-agreement threshold, while agreement on attributes was lower; this measures coding consistency, not correctness or review merit.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.