Reproduce “WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yaoyao Qian, Yuanli Wang, Jinda Zhang, Yun Zong, Meixu Chen, Hanhan Zhou, Jindan Huang, Yifan Zeng and 3 more

WebGraphEval introduces a graph-based evaluation framework for web agents that aggregates thousands of agent trajectories into a unified, weighted action graph, facilitating multi-dimensional analysis of agent behavior and efficiency. This approach uncovers nuanced strategies, common failure points, and critical paths missed by traditional outcome-based metrics.

Key insights

  • Aggregating agent trajectories into a graph representation effectively captures the structural diversity of valid solutions, enabling a deeper understanding of agent strategies and common failure modes.
  • There is a trade-off between an agent's efficiency (measured by action necessity and path optimality) and its overall task success, with different agents exhibiting complementary strengths across varied task categories.
  • LLM-based canonicalization and necessity labeling provide a robust, scalable method for standardizing actions and offering fine-grained insights into trajectory quality, correlating strongly with agent confidence and task complexity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.