Reproduce “Agent Benchmarks Stop Ranking Before They Run Out of Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zihan Dong, Qishi Zhan, Minxuan Hu
The authors propose benchmark scoring that separates validity checks from quality scoring and resource reporting, using dated anchors for measurable objectives. On ALE-Bench’s 40 contests, an objective score yields three tiers among the top ten, while a median-human pass/fail key yields one. In a Terminal-Bench rerun, all models passed one task despite different speedups; results do not establish universal gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.