Reproduce “The BrowserGym Ecosystem for Web Agent Research” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel and 12 more

The BrowserGym ecosystem offers a unified, Gymnasium-compatible framework and accompanying tools (AgentLab) for developing and evaluating web agents across multiple benchmarks. An evaluation of state-of-the-art LLMs and VLMs within this ecosystem demonstrated Claude 3.5 Sonnet's strong performance on enterprise tasks like WorkArena L2 (39.1% success) and GPT-4o's leading visual understanding on VisualWebArena (26.7%).

Key insights

  • Standardizing the web agent environment and tools facilitates the first large-scale, multi-benchmark evaluation, allowing for direct comparison of LLM performance across diverse tasks.
  • Current state-of-the-art LLMs exhibit varied capabilities, with Claude 3.5 Sonnet excelling on enterprise tasks (WorkArena L2) and GPT-4o demonstrating superior multimodal understanding on visual benchmarks.
  • Despite advancements, developing robust and efficient web agents remains a significant challenge due to the complexity of real-world web environments and inherent model limitations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.