Reproduce “OS-ATLAS: A Foundation Action Model for Generalist GUI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding and 3 more
OS-ATLAS introduces an open-source foundation action model capable of understanding graphical user interfaces and executing actions across Windows, macOS, Linux, Android, and web environments. The model outperforms GPT-4 in zero-shot out-of-distribution GUI agent tasks and is built on the largest open-source cross-platform GUI grounding corpus.
Key insights
- Comprehensive, multi-platform GUI-specific training data is essential for developing generalist GUI agents.
- A unified action space effectively resolves action heterogeneity across different platforms, improving multi-task learning and generalization.
- Pre-training on GUI grounding tasks provides a strong foundation for action models, enhancing their ability to understand and interact with diverse interfaces.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.