Reproduce “How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zora Zhiruo Wang, Yijia Shao, Omar Shaikh, Daniel Fried, Graham Neubig, Diyi Yang

The authors introduce a toolkit that converts recorded computer actions into hierarchical workflows, then use it to compare 48 human workers with agents on 16 long-horizon tasks. Agents relied more on programs than visual interfaces, despite some workflow alignment with humans. Average task success was 47.3% for agents and 84.6% for humans; findings are limited to these curated tasks.

Key insights

  • AI agents consistently adopt an overwhelmingly programmatic approach, even for visually-dependent tasks, which significantly contrasts with humans' diverse UI-oriented tool use across occupations.
  • AI tools primarily augment human work by integrating into existing workflows, increasing efficiency by 24.3%, while full AI automation can disrupt and slow human work by 17.7%.
  • Optimal human-agent collaboration and task delegation can be achieved by leveraging human accuracy and nuanced judgment for less programmable tasks, and agent efficiency for readily programmable steps.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.