Reproduce “WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Imene Kerboua, Fatemeh Pesaran Zadeh, Xing Han Lù, Weijian Qi, Alexander Miller, Junyi Song, Yunjia Tian, Dongjin Kang and 14 more
WebArena-Pro evaluates browser agents on 300 human-validated tasks across 20 self-hosted applications, including workflows involving embedded audio and video. It provides transcript and video-frame access and resets seeded application state between tasks. At a 50-action limit, two configurations each achieve 62.3% full-task success, while media tools raise success from 14.0% to 48.8% for one configuration on an audited 43-task subset.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.