Reproduce “GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, Chen Zhao
Researchers from NYU Shanghai, Nanyang Technological University, and Yale NLP Lab conducted a controlled comparison of GUI and skill-mediated CLI computer-use agents, identifying distinct execution bottlenecks for each modality across 440 desktop tasks. The study found that GUI agents generally outperformed original CLI agents, but CLI performance significantly improved with comprehensive skill patching, highlighting the critical role of skill layer design.
Key insights
- Interaction modality fundamentally shapes agent capabilities, with execution logic encoded either within the visual interface (GUI) or through a constructed skill layer (CLI).
- GUI agents are primarily limited by challenges in reliable visual grounding, state tracking over long interaction chains, and accurately executing multi-step workflows.
- CLI agents face significant limitations due to the coverage and scalability of their skill interfaces, with a large portion of performance deficits recoverable through comprehensive skill patching.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.