Reproduce “CUA-Skill: Develop Skills for Computer Using Agent” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyi Chen, Yinheng Li, Michael Solodko, Sen Wang, Nan Jiang, Tingyuan Cui, Junheng Hao, Jongwoo Ko and 7 more
Microsoft researchers developed CUA-Skill, a framework that encodes human computer-use knowledge into reusable, parameterized skills for desktop environments. The accompanying CUA-Skill Agent achieves a 57.5% best-of-three success rate on the WindowsAgentArena benchmark, establishing state-of-the-art performance for computer-using agents.
Key insights
- Encoding human computer-use knowledge into structured, parameterized skills significantly enhances the scalability, reliability, and generalizability of computer-using agents in complex desktop environments.
- A flexible execution graph that unifies GUI primitives and scripts, and incorporates UI predicates, enables skills to adapt to dynamic UI states and reduce brittle behavior, leading to more robust task completion.
- Integrating a reflective memory and a retrieve-augmented LLM planner allows for dynamic skill retrieval, robust argument instantiation, and effective error recovery, consistently improving task success rates across different LLM backbones.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.