Reproduce “StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, Kai Wang
StateM introduces an agent-native control layer that significantly enhances the reliability of long-horizon CLI agents by managing execution states and procedural knowledge externally. The system achieved up to 95.28% raw accuracy on Terminal-Bench 2.1, an increase of 10.38 percentage points, and enabled a 1/11x reduction in total development and evaluation costs for comparable performance using a lower-cost model.
Key insights
- Substantial performance gains in long-horizon agents can be achieved through "harness scaling"—improving the execution system—without modifying the underlying model weights.
- An external, jointly editable control layer like StateM facilitates failure-driven optimization, allowing procedural knowledge to accumulate and persist, improving agent reliability over time.
- The economic efficiency of agent deployment can be significantly improved, as lower-cost models can achieve high performance when paired with an effectively adapted harness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.