Reproduce “SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dayang Liang, Lang Feng, Bo An, Yunlong Liu

SAPO trains a single causal language model to produce policy actions and turn-level state and action values from one rollout per task, avoiding a separate critic and same-task rollout groups. On ALFWorld, SAPO reports 96.1% ± 1.1 success, near PPO’s 95.8% ± 1.2 and above GRPO’s 77.6% ± 5.2. In reported setups, iteration time fell 22.6% and peak memory 35.6%; these are measured cost reductions, not evidence of lower total training compute generally.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.