Reproduce “TTHE: Test-Time Harness Evolution” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, Bo Han

TTHE adapts an LLM agent’s harness on unlabeled evaluation batches, changing its operating procedure without changing model weights. It examines execution traces, proposes and runs edited harnesses, then commits one for the batch and the next adaptation. Across six evaluated benchmarks, it improves over fixed baselines; on BIRD’s selected 50-question hard subset, accuracy is 48.7% versus 12.7%, not full-benchmark performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.