Reproduce “CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yijia Shao, Zora Zhiruo Wang, Neel Ahuja, Yicheng Wang, Bowen Liu, Diyi Yang
CollabSkill evaluates human-agent collaboration on occupational tasks by pairing workers with agents and estimating each partner’s contribution from graded team outcomes. In 386 sessions with 93 U.S.-based Upwork workers, terminal-agent rankings differed from solo rankings: Claude Code rated 74.778 and Codex 71.267 collaboratively, while their solo scores were 77.17 and 78.02. These sample-based estimates do not establish universal workplace performance.
Key insights
- An AI agent's performance in fully autonomous benchmarks does not reliably predict its effectiveness in human-agent collaborative settings.
- Interface design plays a significant role in collaborative agent performance, with graphical interfaces notably enhancing collaboration even with the same underlying LLM.
- Human AI literacy, particularly practical experience and specific 'AI fluency' interaction behaviors, strongly influences the success of human-agent collaboration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.