Reproduce “Mapping global dynamics of benchmark creation and saturation in artificial intelligence” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, Matthias Samwald
Ott et al. introduce systematic methodologies to map the global dynamics of benchmark creation and saturation in Artificial Intelligence, analyzing 3765 benchmarks from Papers With Code. The study reveals rapid saturation in many benchmarks and identifies factors like versatility and dedicated leaderboards as key drivers of benchmark popularity.
Key insights
- AI SOTA trajectories often exhibit periods of stagnation followed by rapid bursts, indicating non-linear progress and potential for unforeseen advancements.
- A small fraction of benchmark datasets accounts for a disproportionately large number of results, highlighting an uneven distribution of research attention.
- Benchmark popularity is strongly correlated with versatility (supporting multiple task types), breadth (multiple sub-benchmarks), the presence of a dedicated leaderboard, and affiliation with top institutions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.