Reproduce “ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning” on OpenResearch

About this paper

Seil Kang, Hangoo Kang, Tarun Suresh, Youngeun Kim, Shreyas Pimpalgaonkar, Seong Jae Hwang, Azalia Mirhoseini

ThunderSyncRL streams learner gradient calculations as trajectory rewards or teacher scores become available, while keeping rollouts and gradients tied to one policy snapshot and applying one optimizer step after the batch completes. In the reported Qwen-3.8-27B GRPO setup, it reaches 74.01% pass@3 on SWE-bench Verified in 121.3 GPU-hours versus 233.5 with Sync, a 1.9× speedup; outcomes depend on workload overlap.

Key insights

• A synchronous optimizer update does not require every computation contributing to it to run synchronously; execution can be pipelined without introducing policy staleness.
• Both GRPO’s group-normalized gradient and OPD’s token-normalized gradient can be decomposed so partial work is safely computed before all batch information is available.
• The speedup depends on how much learner work can overlap with rollouts and on residual learner work at batch completion; the reported method also uses additional memory for partial gradient state.

Summary by alphaXiv.