Reproduce “MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, Shoufa Chen

MobileAgentBench, developed by researchers from Carnegie Mellon University, University of Michigan, Northeastern University, and The University of Hong Kong, introduces a comprehensive and robust benchmark for evaluating mobile LLM agents in realistic Android environments. It utilizes a novel final UI state matching for task success judgment across 100 tasks in 10 applications, revealing that AppAgent achieved the highest overall success rate of 40% among evaluated agents.

Key insights

  • Task success judgment by matching the final UI state, rather than specific action sequences, provides a more robust evaluation method for agents.
  • Agent implementation details and architecture can be more crucial for performance than the underlying Large Language Model (LLM) itself.
  • Access to detailed textual view hierarchy information significantly improves agent performance compared to relying solely on visual information.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.