Reproduce “MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyu Huang, Juyuan Wang, Longfeng Chen, Boyi Xiao, Leng Cai, Yawen Zeng, Jin Xu
Researchers from South China University of Technology and Pazhou Lab developed MVISU-Bench, a comprehensive, user-survey-driven benchmark for mobile agents addressing real-world multi-app, vague, interactive, single-app, and unethical instructions. They also introduced Aider, a plug-and-play module that improved agent success rates by 19.55% overall on the benchmark, with specific gains of 53.52% for unethical tasks and 29.41% for interactive tasks, highlighting severe weaknesses in current agent handling of these critical categories.
Key insights
- MVISU-Bench effectively revealed critical limitations of state-of-the-art mobile agents, particularly their universal failure (0% success) on interactive instructions and very low performance on unethical tasks, while the Aider module provided a practical solution to these challenges.
- Performance discrepancies exist due to language (English instructions yield a 14.16% higher success rate than Chinese) and model type (closed-source generally outperforms open-source), and current mobile agents are too slow (average 39.1 seconds/step) and expensive (average $0.16/task) for widespread real-world adoption.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.