Reproduce “PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar, Firoz Shaik, Shubhang Desai, Thong Q. Nguyen, Muhammad Taqi Raza, Vishal Chowdhary and 1 more
PPT-EVAL introduces a benchmark for computer-use agents performing tasks in Microsoft PowerPoint Online, featuring GUI interaction, 120 diverse tasks, and a detailed rubric-based evaluation. The benchmark reveals that even frontier agents achieve a 45% success rate and 0.57 average partial score, significantly underperforming human users who average 80% success and 0.90 partial score.
Key insights
- A significant performance gap exists between current frontier computer-use agents and human users in GUI-based presentation software tasks, indicating substantial room for improvement.
- GUI-based interaction poses unique challenges for agents, as the API baseline (62% success rate) significantly outperformed the best GUI agent (45% success rate) despite the latter having access to a richer feature set.
- The detailed, partial-credit rubric system provides a reliable and interpretable measure of agent performance for complex, open-ended tasks in multimodal GUI environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.