Reproduce “Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker

Researchers at Cohere and Cohere For AI demonstrate that simpler REINFORCE-style optimization, particularly REINFORCE Leave-One-Out (RLOO), consistently outperforms Proximal Policy Optimization (PPO) and "RL-free" methods for Large Language Model alignment. This approach shows significant win-rate improvements (e.g., 32.1% on HH Llama) while offering greater robustness and computational efficiency by simplifying the problem formulation.

Key insights

  • PPO's variance reduction techniques and clipping mechanisms are often counterproductive in LLM alignment, as strong pre-training leads to a more stable optimization landscape.
  • Treating LLM generation as a sequence-level bandit problem, with rewards attributed to the full sequence, is more effective and computationally simpler than token-level Markov Decision Process modeling.
  • Multi-sample REINFORCE (RLOO) provides superior variance reduction and performance compared to single-sample methods and PPO in the LLM alignment setting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.