Reproduce “MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhao Wang, Mengying Yuan, Zijie Yu, Guangyi Liu, Rui Ye, Tian Jin, Siheng Chen, Yanfeng Wang
MobileA3gent introduces a framework for training mobile GUI agents by utilizing decentralized, self-sourced, and automatically annotated data from diverse users. This approach enables scalable and privacy-preserving agent development, achieving performance comparable to human-annotated methods while reducing annotation costs by up to 99.9%.
Key insights
- Daily mobile phone usage by a global user base represents a vast, untapped, and continuously growing source of real-world supervisory data for training GUI agents.
- Local VLM-based automated annotation with hierarchical reasoning can effectively extract natural language instructions from raw user trajectories, drastically cutting data collection and labeling costs.
- Federated learning can be effectively adapted to privacy-preservingly train VLM-based GUI agents from distributed mobile data by accounting for its unique two-level (episode and step) data heterogeneity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.