Reproduce “Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi and 3 more
Doc2Harness evaluates whether language models can build coding-agent harnesses from behavioral documentation and interface scaffolding. In a two-stage process, each candidate is frozen and driven by a scripted mock model, while tests inspect requests, files, exit codes, and command output. Across 328 scored cases, GPT-6 Astra leads with a 53.05% pass rate but still passes barely half.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.