Reproduce “GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, Liqiang Nie

GUI-explorer introduces a training-free agent that autonomously explores application functionalities and unsupervisedly mines transition-aware knowledge to improve GUI automation. The system notably enhances task success rates on mobile GUI benchmarks and mitigates multimodal large language model misinterpretation of UI components through dynamic, context-specific operational guidance.

Key insights

  • MLLMs exhibit inherent limitations in accurately interpreting UI element functionality and dynamic behavior without external, context-specific guidance.
  • Autonomous exploration and unsupervised knowledge mining can efficiently generate high-quality, transition-aware operational logic without human intervention or prior model training.
  • Dynamically integrating refined, context-specific knowledge significantly enhances MLLM-based GUI agents' ability to interpret UI components and execute tasks accurately, overcoming static knowledge limitations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.