Reproduce “ProBench: Benchmarking GUI Agents with Accurate Process Information” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leyang Yang, Ziwei Wang, Xiaoxuan Tang, Sheng Zhou, Dajun Chen, Wei Jiang, Yong Li
ProBench introduces a mobile benchmark for GUI agents that rigorously evaluates multi-step tasks by accurately capturing intermediate process information in addition to final screen states. Its evaluation across 34 mainstream applications reveals that even state-of-the-art models like Gemini 2.5 Pro achieve average accuracy below 50%, performing significantly worse on process-related tasks and complex application categories.
Key insights
- Evaluating intermediate process information is critical for truly assessing GUI agents, as 'Process-related Tasks' are significantly harder and expose limitations more effectively than final-state evaluations.
- Current state-of-the-art GUI agents universally struggle with fundamental capabilities, including accurate grounding, effective utilization of historical context, and robust complex task planning.
- Agent performance varies considerably across application categories, with higher success rates in static 'Production & Tools' apps but significant difficulties in dynamic 'Social' and 'Lifestyle' applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.