Reproduce “Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimization” on OpenResearch

About this paper

Jianghao Su, Xia Zeng, Luhui Liu, Chao Luo, Ye Chen, Zhuoran Zhuang

This work introduces Progressive Reward Shaping (PRS) and Value-based Sampling Policy Optimization (VSPO) to improve Agentic Reinforcement Learning for Tool-Integrated Reasoning (TIR) tasks. These complementary methods enhance training stability, accelerate convergence, and yield higher performance in LLM-based agents, demonstrating up to a 5.34% increase in Reference Answer Pass Rate for long-form QA and 3.16% in Exact Match for short-form QA over baselines.

Key insights

• Dense, curriculum-inspired reward shaping (PRS) significantly reduces the proportion of zero-advantage tasks and fosters more efficient policy exploration, proving especially beneficial for weaker base models.
• Intelligent sampling based on task value and controlled gradient scaling through value smoothing clipping are essential for overcoming gradient degradation in multi-sample policy optimization, leading to faster convergence and improved training stability.
• Combining a principled reward design with an optimized policy update mechanism allows LLM-based agents to achieve superior performance and better generalization in complex Tool-Integrated Reasoning tasks by improving credit assignment and data utilization.

Summary by alphaXiv.