Reproduce “A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions” on OpenResearch

About this paper

Jianghan Shen, Siqi Luo, Yue Li, Jiyao Liu, Wanying Qu, Yi Zhang, Ziyan Huang, Tianbin Li and 4 more

The authors survey policy-gradient methods for language models through the two factors in expected reward: sampled trajectories and assigned rewards. They trace methods from REINFORCE and PPO to GRPO and extensions, explaining how GRPO replaces critic-based advantages with normalized rewards across responses to one prompt while retaining PPO’s ratio, clipping, and KL regularization. The framework organizes failure modes but is not experimentally validated.

Key insights

• All LLM policy gradient algorithms optimize the same fundamental expected reward objective, with their distinct properties arising from targeted interventions on either the trajectory generation or the reward evaluation components.
• The two-axis framework provides a diagnostic lens to mechanistically understand the causes of algorithmic failures (e.g., sparse rewards, unstable updates) and to precisely pinpoint where existing methods intervene.
• Many complex instabilities and advanced LLM policy optimization paradigms, such as Agentic RL and On-Policy Distillation hybrids, involve coupled failures that necessitate joint interventions across both the trajectory and reward sides.

Summary by alphaXiv.