Reproduce “PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost” on OpenResearch

About this paper

Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang, Ritu Gala, Charles Wang, Sugam Dipak Devare, Khushi Bhardwaj, Abhibha Gupta and 4 more

PivotRL trains agentic policies by selecting difficult decision points from expert trajectories and sampling short, on-policy actions there, with reward for locally acceptable alternatives. Across four domains, it gains an average 14.11 percentage points over the base model, versus 9.94 for same-data SFT, while average OOD change is +0.21 points; it still trails SFT on SWE-Bench Verified.

Key insights

• Only groups of rollouts with positive reward variance (mixed success and failure) contribute non-zero group-normalized advantage updates, justifying the pivot filtering strategy.
• The strength of local learning signals along the KL path is directly proportional to the reward standard deviation, indicating that turns with mixed outcomes provide richer training gradients.
• Functional reward-based RL, when combined with KL regularization, enables the policy to shift probability mass towards acceptable actions while maintaining the relative ordering of non-acceptable actions, thus mitigating OOD degradation.

Summary by alphaXiv.