Reproduce “All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning” on OpenResearch

About this paper

Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, J. Andrew Bagnell

Researchers explain the empirical superiority of two-stage online Reinforcement Learning from Human Feedback (RLHF) over offline Maximum Likelihood Estimation (MLE) for fine-tuning foundation models. This advantage stems from a “generation-verification gap,” where learning a simpler reward model (verifier) is statistically more efficient than directly learning a complex policy (generator), effectively constraining the policy search space.

Key insights

• Under ideal theoretical conditions (isomorphic function classes, exact optimization), online and offline PFT methods are mathematically equivalent and should yield identical optimal policies.
• The observed empirical superiority of online RLHF for LLM fine-tuning is primarily explained by a 'generation-verification gap,' where the optimal reward function (verifier) is often statistically simpler to learn than the optimal policy (generator).
• Online PFT implicitly constrains the search space to policies optimal for these simpler verifiers, effectively reducing the statistical complexity of the learning problem compared to offline methods that search the entire policy space.

Summary by alphaXiv.