Reproduce “HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tongbo Chen, Junbo Niu, Zhengxi Lu, Niu Lian, Fei Tang, Yuchen Yan, Yike Hong, Yong Du and 3 more
HybridCUA trains computer-use agents to choose between GUI actions and shell commands, rather than assuming shell access improves performance. It combines GUI-only, CLI-only, and interleaved demonstrations with task-level interface-choice rewards and command-execution penalties. On OSWorld, HybridCUA-9B reaches 53.6% accuracy in 14.0 steps, versus 50.4% and 22.1 for GUI-only training, under the tested benchmark conditions.
Key insights
- Interface choice is itself a learnable behavior: CLI actions are favored for bulk editing, setup, file manipulation, and verification, while GUI actions remain preferable for spatially grounded operations and visual adjustment.
- Mixed GUI and CLI supervision outperforms training on either modality alone; SFT reaches 46.0% on OSWorld with combined data versus 43.2% for GUI-only and 31.7% for CLI-only data.
- The two CLI-specific rewards address different failure modes: trajectory-level supervision improves efficiency and usage calibration, whereas step-level feedback lowers command execution errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.