Reproduce “Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiyuan Jiang, Shenghao Xie, Wenyi Li, Wenqiang Zu, Peihang Li, Jiahao Qiu, Siqi Pei, Lei Ma and 3 more

Researchers from Peking University, Princeton University, and other institutions developed ZoomClick, a training-free inference strategy that enhances GUI grounding accuracy by dynamically zooming into relevant regions, and GUIZoom-Bench, a diagnostic benchmark for evaluating zoom behaviors. ZoomClick achieved a new state-of-the-art of 73.1% accuracy on ScreenSpot-Pro and a 66.7% relative improvement on UI-Vision, enabling smaller models to outperform larger unaugmented counterparts.

Key insights

  • A carefully engineered, training-free zoom strategy can leverage models' inherent spatial priors to significantly improve localization accuracy, even allowing smaller models to surpass larger unaugmented ones.
  • Effective zoom requires a reliable initial localization (pre-zoom) and gradual, iterative refinement (multi-step narrowing) rather than single, deeper shrinks, as well as an appropriate minimum crop size to retain essential context.
  • Models struggle to interpret inference-time injected context (e.g., explicit descriptions or visual markings) across zoom rounds, performing best with no explicit context injection, suggesting a need for zoom-aware training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.