Reproduce “Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyan Wang, Zheng Wang, Xingwei Qu, Qi Cheng, Jie Fu, Shengpu Tang, Minjia Zhang, Xiaoming Huo
The Slow-Fast Policy Optimization (SFPO) framework, developed by researchers from Georgia Institute of Technology, University of Illinois Urbana-Champaign, and NVIDIA, introduces a three-stage update mechanism for on-policy reinforcement learning that stabilizes gradient estimates and reuses rollout data more effectively in LLM reasoning tasks. It achieves an average accuracy improvement of up to 2.80 points on math reasoning benchmarks and reduces required rollouts by up to 4.93 times, resulting in up to 4.19 times faster training compared to vanilla GRPO.
Key insights
- Performing multiple inner gradient updates on a single rollout batch can effectively damp idiosyncratic fluctuations and produce a more stable update direction.
- Interpolating the updated parameters back towards the original policy (repositioning) acts as a regularizer, preventing off-policy drift and maintaining training stability, especially when reusing data.
- An adaptive scheduling of the interpolation parameter α, informed by policy entropy, allows for aggressive exploitation early in training while prioritizing stability near convergence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.