Reproduce “Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak and 3 more
Cohere researchers developed Contrastive Policy Gradient (CoPG), a reinforcement learning algorithm for aligning Large Language Models to arbitrary sequence-level reward functions in a supervised-friendly, off-policy manner. This approach achieves higher reward scores and a more favorable KL-reward trade-off compared to existing direct alignment methods.
Key insights
- CoPG's objective function possesses a unique minimizer that corresponds to the optimal policy of the underlying KL-regularized RL problem, providing strong theoretical guarantees for its convergence.
- The algorithm offers a mathematically principled off-policy policy gradient that avoids the high variance associated with importance sampling, a common challenge in traditional off-policy RL.
- CoPG serves as a unifying framework, demonstrating that it generalizes classical policy gradient, Reinforce Leave-One-Out (RLOO), and, when rewards are binarized, Identity Preference Optimization (IPO).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.