Reproduce “Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré
The authors describe a recurring process for auditing and repairing agent benchmark tasks using reports and traces, agent-proposed diagnoses, reference-solution checks, and human review. In Terminal-Bench 2.0, they revised 28 of 89 tasks. Across 19 agents, scores changed by up to 12 percentage points, while rankings remained highly correlated (Spearman rho = 0.958); close rankings could still shift.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.