Reproduce “Life After Benchmark Saturation: A Case Study of CORE-Bench” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nitya Nadgir, Sayash Kapoor, Kangheng Liu, Peter Kirgis, Matilda Orona, Stephan Rabanser, Tilman Bayer, Abhishek Shetty and 6 more
The paper argues benchmark accuracy saturation does not mean a benchmark has nothing more to reveal. The authors audit CORE-Bench Hard through task-level outcomes, validity, reliability, agent scaffolds, and human–agent work. They find equal scores can hide different solved tasks, while collaborative sessions lasted 2.11 times shorter than manual sessions. This case study does not establish correctness or general productivity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.