Reproduce “Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning” on OpenResearch

About this paper

Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang and 3 more

Group Relative Reward Rescaling (GR3) provides a framework for training Large Language Models with Reinforcement Learning that prevents them from generating unnecessarily long responses, a problem known as length inflation. It achieves this by adaptively rescaling rewards, leading to improved task performance and significant token savings (e.g., over 40% reduction in mathematical reasoning tasks) across both RLHF and RLVR settings.

Key insights

• Multiplicative reward rescaling directly links length control to task reward, preventing the compensatory effects seen with additive penalties where models might prioritize brevity over quality.
• Dynamic length budgeting through group-relative regularization, rather than static thresholds, allows for adaptive control that avoids suppressing necessary reasoning or introducing off-policy bias.
• By promoting conciseness, the framework enhances the learning signal's density by focusing on causally important tokens, often leading to better task performance than standard, unconstrained RL.

Summary by alphaXiv.