Reproduce “MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu, Yinghao Ma, Jyh-Shing Roger Jang, Hung-yi Lee
MetaBench-Harness evolves the workflow for generating and evaluating benchmark tasks using prior code, scores, and execution traces. In CodeContests, repairing an answer-format evaluator raised Claude-Sonnet-5’s score from 0.300 to 0.948 without changing task difficulty. On AIME-2024, DeepSeek-V3.1 scored 0.322 with MetaBench-Harness versus 0.817 with a fixed harness, supporting harness evolution in this setup, not universal prevention of benchmark saturation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.