Reproduce “AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongxin Li, Jingfan Chen, Jingran Su, Yuntao Chen, Qing Li, Zhaoxiang Zhang
Researchers introduced AutoGUI, an automated pipeline leveraging large language models to generate contextual functionality annotations for UI elements at scale. This yielded the AutoGUI-704k dataset, which improved VLM UI grounding accuracy by up to 26.3% on benchmarks like FuncPred, achieving 96.7% annotation correctness.
Key insights
- LLMs can effectively infer complex contextual functionality of UI elements by interpreting UI state changes, providing richer semantics than raw code or brief labels.
- Integrating LLM-aided rejection and verification processes is crucial for automated annotation, significantly improving annotation correctness from 64.5% to 96.7%.
- Detailed contextual functionality annotations are more effective for training robust UI grounding models compared to simpler element-HTML pairs or condensed descriptions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.