Reproduce “OpenComputer: Verifiable Software Worlds for Computer-Use Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinbiao Wei, Qianran Ma, Yilun Zhao, Xiao Zhou, Kangqi Ni, Guo Gan, Arman Cohan
OpenComputer is a framework that generates verifiable software worlds for computer-use agents by integrating automated task generation with programmatic state verification. This approach creates a benchmark of 1,000 tasks across 33 desktop applications, revealing that frontier agents achieve an average reward of 88.4% and a 68.3% success rate while demonstrating the higher reliability of hard-coded verifiers over LLM-as-judge methods.
Key insights
- Programmatic, verifier-grounded evaluation achieves higher alignment with human adjudication (97.3% agreement) compared to LLM-as-judge methods (92.2% agreement), especially for fine-grained application states or non-visual success criteria.
- A self-evolving verification layer can effectively identify and repair checker-side errors, achieving an 89.4% repair rate on identified issues and significantly improving verifier reliability.
- Performance on existing desktop benchmarks does not reliably generalize to the broader and more heterogeneous software settings of OpenComputer, highlighting persistent limitations in current agent capabilities for robust computer automation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.