Reproduce “Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song and 2 more

FOCUS introduces a framework for GUI grounding that employs a dual-system cognitive approach, combining fast direct prediction with deliberate reasoning steps. This enables superior performance on challenging interfaces, particularly for icon and widget elements, by adaptively balancing efficiency and thoroughness.

Key insights

  • A dual-system cognitive architecture, combining fast intuitive responses with slower, deliberate reasoning, can significantly enhance GUI grounding performance.
  • Explicitly structuring reasoning chains into interface summarization, visual focused analysis, and precise coordinate prediction via data synthesis improves multimodal model capabilities.
  • An adaptive switching mechanism allows the system to dynamically balance between efficiency and thoroughness based on task complexity, leading to an optimal trade-off.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.