Reproduce “GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization” on OpenResearch

About this paper

Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin and 5 more

The GDPO method trains language models with multiple rewards while preserving distinctions among reward combinations. It normalizes each reward within a response group, combines the signals, then normalizes advantages across the batch. On DeepSeek-R1-1.5B, it reaches 29.4% AIME Pass@1 accuracy versus GRPO’s 23.1%, with 6.5% of responses exceeding 4,000 tokens; it remains below the original model’s 29.8%.

Key insights

• Summing individual rewards before group-wise normalization in algorithms like GRPO causes a loss of information (reward signal collapse), leading to a less effective training signal.
• Decoupling the normalization process for each reward component preserves a significantly richer and more expressive advantage signal, crucial for accurate multi-objective optimization.
• Simply weighting rewards is often insufficient for guiding model behavior when objectives differ greatly in difficulty; instead, conditioning an easier reward on a harder one is more effective for prioritized learning.

Summary by alphaXiv.