Reproduce “GTA: Generating Long-Horizon Tasks for Web Agents at Scale” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu

The GTA framework from researchers at Jilin University and the Institute of High Energy Physics, CAS, introduces a scalable, automated pipeline for generating long-horizon, multi-hop web-agent tasks directly from live web content. This system drastically reduces the cost and time associated with creating complex benchmarks, revealing significant performance gaps for current web agents, which achieve less than 30% success on these new, challenging tasks.

Key insights

  • Decoupling the computationally intensive crawling and site graph construction from LLM-based task generation drastically improves efficiency, reducing the marginal cost for additional tasks to approximately $0.10 per task.
  • Combining retrieval-based seeding with in-context generation enables broader exploration of website functionalities and naturally fosters compositional, multi-hop tasks by forcing reasoning across distinct web elements.
  • A multi-stage LLM-based verification process is essential for maintaining high quality, solvability, and reproducibility of automatically generated complex web tasks, safeguarding against common issues in purely generative approaches.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.