Reproduce “A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents” on OpenResearch

About this paper

Ananya Gubbi Mohanbabu, Rosiana Natalie, Brandon Kim, Anhong Guo, Amy Pavel

Researchers from the University of California, Berkeley and the University of Michigan developed the A11y-CUA dataset, which captures multimodal computer interaction traces from both sighted and blind/low-vision users. This work quantifies distinct interaction styles between user groups and demonstrates that state-of-the-art Computer Use Agents exhibit significantly reduced task success rates, falling from 78.33% to as low as 28.33%, when operating under assistive technology conditions.

Key insights

• Blind and low-vision users (BLVUs) exhibit distinct, keyboard-dominant interaction patterns, taking 2.28 times longer and executing more than twice the number of keyboard actions per task compared to sighted users (SUs).
• Significant interaction diversity exists within both user groups, with BLVUs employing varied keyboard-based strategies such as 'walking', 'chunk-jumping', and 'ribbon/OS routes'.
• CUAs face specific accessibility gaps (perception, cognitive, action) when operating under AT conditions, struggling with off-screen controls, multi-step task tracking, and efficient input methods like hotkeys.

Summary by alphaXiv.