Reproduce “When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav and 29 more

The authors measure whether leading language models remain distinguishable on text-only benchmarks by comparing the spread of the five highest scores with estimated evaluation uncertainty. Their saturation index is higher when top scores cluster more tightly relative to uncertainty; its ceiling uses the highest observed score and does not establish task mastery. In 60 benchmarks, 29 had high or very high saturation, but age associations were not statistically significant, and results are cross-benchmark associations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.