Reproduce “AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu
AgentTrek, developed by researchers from the University of Hong Kong and Salesforce Research, introduces an automated pipeline that synthesizes high-quality, multimodal agent trajectories by leveraging web tutorials and guided replay. This approach yields a comprehensive dataset for training GUI agents, achieving state-of-the-art performance on text-based and vision-based web browsing benchmarks while significantly reducing data collection costs.
Key insights
- Publicly available web tutorials can be effectively transformed into structured, executable task specifications, serving as a scalable source of human-curated knowledge for agent training.
- A guided replay mechanism in real web environments, coupled with VLM-based evaluation, enables the cost-effective synthesis of high-quality, multimodal trajectories without requiring human annotators.
- Integrating both textual (accessibility trees) and visual (screenshots, pixel-level actions) modalities within synthesized trajectories is crucial for training GUI agents with robust visual grounding and multi-step planning capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.