Reproduce “H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Solim LeGris, Wai Keen Vong, Brenden M. Lake, Todd M. Gureckis
Researchers at New York University provide the first comprehensive estimate of human performance on the full Abstraction and Reasoning Corpus (ARC) benchmark, reporting 76.2% accuracy on training tasks and 64.2% on evaluation tasks. This work establishes a robust human baseline against which AI systems can be measured, revealing a substantial gap in abstract reasoning capabilities, and offers the publicly available H-ARC dataset of human solutions and behaviors.
Key insights
- Human competence on ARC is remarkably high, with 98.8% of all 800 tasks solvable by at least one participant, and over 90% solvable by at least three.
- The evaluation set of ARC tasks is significantly harder for humans than the training set, not primarily due to larger grid sizes but likely due to more complex underlying rules.
- Humans and AI models make substantially different types of errors, and humans show superior self-correction capabilities, significantly improving accuracy with additional attempts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.