Reproduce “SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu and 1 more
SPPO trains reasoning models with verifiable rewards by treating each response as one action and using a prompt-level value model to estimate success. It compares that estimate with the outcome and broadcasts one advantage across response tokens, avoiding group sampling. On five math benchmarks, its 7B policy with a 1.5B critic scores 58.56 versus GRPO’s 57.44, but not on every benchmark.
Key insights
- The effectiveness of existing group-based methods implicitly demonstrates the structural robustness gained by modeling reasoning at the sequence level.
- Estimating the scalar solvability of a problem is less complex than token-level generative reasoning, allowing for a smaller, decoupled critic model to align larger policies.
- Uniformly applying a single sequence-level advantage to all tokens in a response circumvents the temporal credit assignment problem in sparse-reward, long-horizon tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.