Reproduce “AutoGUIWorld: Image Generators as Visual World Models for GUI Agent” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng and 13 more
AutoGUIWorld generates GUI training trajectories by planning actions for invented scenes and using image editing to depict each next screenshot state, without running the depicted applications. Its filtered dataset contains 79,266 step-level samples across four operating systems. Fine-tuning Qwen3.5-35B-A3B raises OSWorld’s mean task score from 33.0% to 40.8%, but these are visual-state trajectories, not executions of underlying software.
Key insights
- Useful training experience can be synthesized at the screenshot level without recreating an application’s internal state, and the resulting data can transfer to real desktop and scientific-software benchmarks.
- Visual plausibility is not enough to ensure interaction fidelity: audits found residual mismatches, especially for dragging, scrolling, and text entry, while generated screenshots remained distinguishable from real data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.