Reproduce “GUICourse: From General Vision Language Models to Versatile GUI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu and 6 more
GUICourse introduces a comprehensive pipeline and diverse datasets to train general Vision Language Models into versatile GUI agents. The approach enables a 3.1B parameter agent (MiniCPM-GUI) to achieve a 57.1% mean Step Success Rate across web and smartphone tasks and improves performance on external benchmarks like Mind2Web by 2-10 points over baseline VLMs.
Key insights
- Enhancing fundamental OCR and visual grounding capabilities in VLMs using targeted large-scale data (GUIEnv) is a necessary prerequisite for effective GUI navigation performance.
- A comprehensive, multi-stage data-driven training approach can transform general VLMs into versatile GUI agents, enabling even smaller models (e.g., 3.1B parameters) to perform comparably to larger ones.
- Integrating conversational data (GUIChat) not only improves natural language interaction but also contributes to better GUI navigation performance, suggesting a holistic improvement in understanding and reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.