Reproduce “ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianchen Guan, Xinlei Lin, Royce Cheng-Yue, Xiangjun Wang, Shuyan Zhou
ComponentBench evaluates short, realistic web tasks centered on UI components, verifying committed end states and comparing agents with human reference traces. Its Full suite contains 2,910 tasks across 97 component types and 14 families. GPT-5 mini passes 83.1% with AX-tree observations but 48.9% with Pixel input. The benchmark covers three React libraries and does not test long-horizon planning.
Key insights
- The choice of observation and action space significantly impacts agent performance, with structured aids often benefiting weaker models while stronger models can sometimes perform comparably or even better in raw pixel mode.
- Agents demonstrate a substantial efficiency gap compared to humans, consistently requiring more steps and time to complete tasks, even when eventually successful.
- UI components that involve precise spatial manipulation, often trivial for humans to complete in one or two steps, pose disproportionately high challenges for computer-use agents.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.