Reproduce “APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi and 2 more
AppSim-Bench introduces a framework for reproducible evaluation of mobile GUI agents, constructing simulated real-world apps with deterministic outcome verification to bridge the gap between realistic tasks and consistent benchmarking. The evaluation of 19 state-of-the-art agents reveals that even the best models achieve only about 50% accuracy on diverse tasks, particularly struggling with long-horizon workflows and numerical reasoning.
Key insights
- Autonomous mobile GUI task execution remains largely unsolved for state-of-the-art agents, with the top model (Claude-Opus-4.7) achieving only 50.27% overall accuracy.
- Agent performance consistently degrades with increasing task length, indicating a major bottleneck in handling long-horizon multi-step workflows and accumulating errors.
- Failures in numerical reasoning tasks, particularly threshold filtering, primarily stem from reasoning difficulties rather than perception or extraction errors, highlighting a need for improved quantitative logic.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.