Reproduce “REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jian Hu, Jason Klein Liu, Haotian Xu, Wei Shen
REINFORCE++ trains language models with preference feedback without a separately trained critic, replacing prompt-local reward scaling with statistics computed across the training batch. Its baseline variant also subtracts the mean reward within each prompt’s response group. On five tool-use benchmarks, that recipe reports an average@32 accuracy of 24.10, versus 22.58 for GRPO, but scores lower than GRPO on AIME 2024.
Key insights
- Prompt-level (local) advantage normalization, as used in prior critic-free methods, is a mathematically biased estimator due to non-independent numerator and denominator.
- Global Advantage Normalization provides a stable and effectively unbiased gradient estimate, especially as batch sizes increase, preventing 'advantage explosion' and improving generalization.
- The correct choice of KL regularization estimator, specifically the k2 estimator for Reverse KL divergence, is crucial for stable and theoretically sound critic-free RLHF.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.