Reproduce “SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bingshuai Liu, Ante Wang, Zijun Min, Liang Yao, Haibo Zhang, Yang Liu, Xu Han, Peng Li and 2 more
SPEC-RL introduces a method to accelerate on-policy Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs) by integrating speculative decoding. This approach reuses cached rollouts from previous training epochs, achieving a 2-3x speedup in rollout generation and reducing token generation by approximately 66% without compromising policy quality across various LLM backbones and RL algorithms.
Key insights
- Rollouts generated across consecutive RLVR training epochs exhibit substantial overlapping segments due to incremental policy updates, representing a significant source of computational waste.
- Speculative decoding can be effectively adapted for RL training by leveraging the previous policy's cached outputs as an implicit draft model, eliminating the need for an auxiliary draft model.
- Introducing a moderate 'lenience' in the speculative verification process can significantly boost rollout efficiency (2-3x speedup) without negatively impacting policy learning stability or performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.