Reproduce “WebSuite: Systematically Evaluating Why Web Agents Fail” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Li, Jim Waldo
Harvard University researchers introduced WebSuite, the first diagnostic benchmark for generalist web agents, which systematically evaluates agent failures by attributing them to specific interaction types within a multi-tiered taxonomy. Experiments with popular agents revealed distinct weaknesses, such as a 0% success rate for SeeAct on 'select' interactions and natbot struggling with complex form filling, providing actionable insights for targeted improvements.
Key insights
- A multi-layered taxonomy, adapted from digital literacy research and modern web standards, provides a robust framework for systematically decomposing and analyzing agent interactions.
- Attributing end-to-end task failures to specific interaction failures offers precise, actionable feedback for agent development, a capability missing in traditional benchmarks.
- Operational failures (basic UI interactions) and informational failures (reasoning and data manipulation) often compound, emphasizing the need to diagnose fundamental interaction issues to improve higher-level task performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.