Reproduce “AI and the Everything in the Whole Wide World Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, Alex Hanna
This paper critically examines the limitations of influential AI benchmarks, arguing that they fail to accurately measure general capabilities due to issues with construct validity, de-contextualized data, and inappropriate community use. It highlights how current benchmark practices can mislead research and justify unverified claims of AI progress, advocating for a re-evaluation of how AI systems are assessed.
Key insights
- Benchmarking "general" AI capabilities using limited, specific datasets is fundamentally flawed, analogous to trying to contain "everything in the whole wide world" within a museum.
- The Common Task Framework (CTF) has been inappropriately extended from tightly-scoped, practical tasks to abstract, general objectives, distorting its original utility.
- Current benchmark-driven "state-of-the-art chasing" practices lead to intellectual dishonesty, misdirected research, and can enable the unjustified deployment of potentially harmful systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.