Reproduce “GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xinyu Geng, Yanjing Xiao, Yuyang Zhang, Hanwen Wang, Xinyan Liu, Rui Min, Tianqing Fang, Yi R. Fung

Hong Kong University of Science and Technology introduces GeoBrowse, a geolocation benchmark designed to evaluate multimodal large language models' agentic tool use by combining weak visual cues with multi-hop knowledge verification. The framework, GATE, enhances models like GPT-4o, increasing accuracy from 23.1% to 31.8% on visual tasks and 11.9% to 21.8% on knowledge-intensive tasks, with stronger models achieving better results through adaptive tool orchestration.

Key insights

  • Agentic planning and dynamic tool orchestration significantly improve multimodal agent performance, outperforming direct inference and fixed tool-use policies, especially in complex multi-hop reasoning tasks.
  • The effectiveness of tools is task-dependent, with visual preprocessing tools being dominant for image-centric tasks and web search/browsing tools critical for knowledge-intensive verification, highlighting the need for flexible tool-use strategies.
  • Analysis of reasoning traces reveals distinct failure modes: strong models often struggle with synthesizing retrieved evidence and making final decisions, while weaker models frequently fail at early perception, grounding, or forming effective queries.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.