Reproduce “Effective Harness Engineering for Algorithm Discovery with Coding Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yoichi Ishibashi, Taro Yano, Masafumi Oyamada
This research demonstrates that improvements to the 'harness,' the underlying infrastructure guiding coding agents, are critical for AI-driven algorithm discovery, often surpassing the impact of raw language model capability. The Vesper framework introduces autonomous coding agents and hack detection, validating that deeper reasoning per algorithm is more cost-effective than generating numerous, brief attempts, achieving state-of-the-art results for the Circle Packing problem at low monetary cost.
Key insights
- Investing more tokens for deeper reasoning per algorithm is empirically more budget-efficient and leads to superior algorithm discovery performance compared to generating many algorithms with brief thought.
- The integration of autonomous coding agents and their multi-step reasoning capabilities significantly improves discovery performance, often more so than simply scaling the underlying LLM's raw power.
- More capable LLMs are more prone to generating evaluation hacks, highlighting the increasing necessity of robust hack detection mechanisms in advanced discovery systems.
- Git worktree isolation provides an effective and scalable solution for enabling stable parallel execution of multiple coding agents with direct filesystem access.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.