Reproduce “GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Li, Yuchen Liu, Haoyu Lu, Zhiqiang Xia, Hongzhen Wang, Kaiyang Han, Changpeng Yang, Jinyang Wu and 3 more
Researchers from Xiaomi Corporation developed GUI-CEval, the first comprehensive benchmark for mobile GUI agents specifically in Chinese environments. It evaluates agents across five core dimensions using real devices and mainstream Chinese applications, revealing that leading MLLMs achieve a 61.41% average score but struggle significantly with online real-world tasks, peaking at 33.33% success.
Key insights
- Leading MLLMs, including domain-specialized ones, show significant weaknesses in reflective decision-making, error identification, and assessing task completion, with peak scores of 21.01% for Reflection tasks.
- Online real-world task performance is substantially lower than foundational or offline tasks, rapidly declining for longer task sequences and when initial states deviate from home screens.
- While larger models offer improvements, task-specific fine-tuning is critical for competitive results, underscoring the importance of targeted training strategies over mere parameter scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.