Reproduce “ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guohong Liu, Jialei Ye, Pengzhi Gao, Wei Liu, Jian Luan, Yunxin Liu, Yuanchun Li
ScaleWoB presents a framework utilizing coding agents to synthesize interactive, backend-free web GUI environments for evaluating AI agents across mobile, desktop, and automotive platforms. The system automatically generates verifiable reward functions for diverse tasks, revealing that state-of-the-art GUI agents achieve an average success rate of 27.92% compared to 92.08% for humans, and significantly underperform on long-horizon and information-intensive tasks.
Key insights
- Leveraging coding agents for environment construction can drastically reduce the engineering cost and increase the scale of interactive, high-fidelity GUI benchmarks.
- Co-generating verifiable reward functions with the environments themselves eliminates the need for unstable LLM-as-judge methods or labor-intensive manual rule engineering.
- State-of-the-art GUI agents exhibit significant performance gaps compared to humans, especially in long-horizon reasoning, tasks requiring structured information returns, proactive exploration, and fine-grained interaction control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.