Reproduce “OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rui Yang, Qianhui Wu, Yuxi Chen, Hao Bai, Wenlin Yao, Hao Cheng, Baolin Peng, Huan Zhang and 2 more

OpenWebRL trains a visual web agent through live-task attempts, using a small supervised warm start, textual interaction history, browser feedback, and multimodal multi-turn GRPO with trajectory-level rewards. OpenWebRL-4B, initialized with 412 supervised trajectories and about 2.2K online tasks, reaches 67.0% on Online-Mind2Web and 64.0% on DeepShop, without showing reduced training compute.

Key insights

  • Both supervised fine-tuning (SFT) and subsequent online MM-GRPO training are crucial, with SFT providing a vital warm-start that leads to faster and more effective RL.
  • Textual environment feedback and historical reasoning traces are critical for online RL performance, providing explicit action outcomes and a compact memory for long-horizon planning, respectively.
  • A distilled 8B judge model offers a cost-effective and equally performant alternative to proprietary large language models for generating reliable trajectory-level reward signals.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.