Reproduce “Self-Harness: Harnesses That Improve Themselves” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, Shuyue Hu
Self-Harness enables a fixed language model to improve its agent harness—prompts, tools, runtime rules, and checks—by using failure traces to propose bounded edits. It accepts candidates only when held-in and held-out regression checks show no degradation. Across nine model–benchmark combinations, pass rates improved, including GLM-5 on AppWorld from 44.4% to 85.0%, within fixed benchmark splits.
Key insights
- A fixed language model can significantly enhance its capabilities by iteratively modifying its own surrounding operational infrastructure (harness) without altering its internal parameters or relying on external human guidance or more capable agents.
- The self-improvement mechanism generates model-specific harness adaptations, addressing unique behavioral patterns, tool-use habits, and error modes inherent to individual LLMs.
- The robust validation process, including regression testing against held-out data, ensures that accepted harness modifications yield reliable and generalizable performance improvements without introducing new issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.