Reproduce “Rethinking Verification in Agent Harnesses” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengxin Zhang, Erkang Zhu, Yin Lin, Claire Cardie, Bolin Ding, Jingren Zhou
The paper compares verification designs in AI coding-agent harnesses, including a separate critic that tests the deliverable and returns findings for the agent to repair. Across evaluated designs, independent review raises average accuracy by 5.9 to 9.5 percentage points on Terminal-Bench and 3.9 to 6.6 on SWE-bench Verified, at modest added priced cost; results cover selected benchmarks, not all tasks.
Key insights
- Verification outcomes depend on intervention timing and reviewer identity; self-check prompts and reminders provide modest, model-dependent gains, while rejecting completion can improve accuracy at substantially higher cost.
- Independent review offers the strongest accuracy–cost tradeoff among the tested coding designs, and more review rounds or critics can improve accuracy further.
- More context is not always better: changes-based review outperformed trajectory-based review on VisualBench, while explicitly asking critics to inspect rendered or interactive results improved preference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.