Reproduce “One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siqi Yang, Qianlan Yang, Yu-Xiong Wang, Saurabh Pujar, Martin Hirzel
A study systematically explores how self-evolving harnesses improve coding agents, revealing that gains primarily compensate for execution defects rather than extending core capabilities. The research found that while abstract disciplinary concepts are portable, 20-40% of evolved harness content is ecosystem-specific, impacting transferability across different programming languages and models.
Key insights
- Self-evolving harnesses primarily compensate for "harness-recoverable execution defects" (e.g., editing test files, shipping non-compiling code) rather than extending the base model's fundamental capabilities.
- Evolved harnesses encode abstract disciplinary concepts that are moderately shared across languages, but their concrete ecosystem-specific instantiations are largely disjoint, with 20-40% of content being ecosystem-specific.
- Gains from self-evolution were minimal in "null regions" (Python across models, GPT-5-mini across languages) where base models already exhibited few harness-recoverable defects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.