Reproduce “On the Reliability of Computer Use Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gonzalo Gonzalez-Pumariega, Saaket Agashe, Jiachen Yang, Ang Li, Xin Eric Wang
The authors evaluate repeated-run reliability in OSWorld, comparing success on at least one of ten attempts with success on all ten. Agent S3 with GPT-5 reaches about 78% success on at least one attempt, but about 36% on all ten. Clarification feedback outperforms binary retry, while planning gains vary by model; conclusions are limited to these benchmark conditions.
Key insights
- Instruction ambiguity is a primary driver of agent unreliability, with clear, unambiguous task specifications significantly improving consistent performance.
- Agent planning variability across repeated executions is a substantial source of inconsistency, which can be mitigated through structured feedback and iterative plan refinement.
- Eliminating stochasticity in LLM decoding does not inherently guarantee reliability and can sometimes hinder performance, suggesting a complex interplay of factors affecting consistent behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.