Reproduce “Locating Hidden Failures Makes Long-Horizon Agents More Reliable” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Salman Rahman, Yubin Kim, Mihir Parmar, A. Ali Heydari, Genglin Liu, Simon A. Lee, Weizhi Zhang, Arian Hosseini and 10 more
The authors label first mistakes in 2,518 agent trajectories across software engineering, computer use, and science, then train Scout, a 4B-parameter verifier, to locate consequential errors and rank candidate runs. In a fixed-pool test on Terminal-Bench 2.0, Scout-based best-of-N selection reached 90.2% success versus 81.8% pass@1 for Claude Opus 4.6, demonstrating selection gains rather than live correction or harm prevention.
Key insights
- Reasoning and planning errors are more often root faults than visible action errors: they account for roughly 46% of first mistakes, while action and execution errors comprise about 24%.
- Once an error occurs, the probability of another wrong step rises sharply—from 3.0% to 40.4% in software engineering and from 5.3% to 58.1% in computer use—and 69.5% of runs fail to recover.
- A locally correct final action is a poor reliability signal: 84.1% of failed software-engineering and computer-use runs end with a step that appears correct, while many unsafe actions are irreversible and missed by simple command-pattern scans.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.