Reproduce “Design and Analysis of the NIPS 2016 Review Process” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, Ulrike von Luxburg

The authors analyze NIPS 2016 peer review and assess whether reviewers’ rankings reveal distinctions hidden by feature scores. After reviews, 2,189 reviewers ordered their papers, but rankings did not affect decisions. More than 30% of comparisons tied on each feature, while median scores agreed with rankings about 90% of the time. The post-hoc study establishes a diagnostic, not improved decisions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.