Reproduce “Into the Unknown: Benchmarking AI agents on open scientific research problems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack and 20 more
Into the Unknown benchmark evaluates agents pursuing scientist-proposed follow-up research using a paper, code, and data. Agents return a paper, code, and logs, graded for integrity, completeness, and meeting the scientist’s success conditions. Across 135 submissions on 15 computational problems, none passed all three strict gates, while 114 failed integrity. These results assess stated requirements, not scientific merit.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.