Reproduce “WPO: Enhancing RLHF with Weighted Preference Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu

Weighted Preference Optimization (WPO) enhances Reinforcement Learning from Human Feedback (RLHF) by intelligently reweighting off-policy preference pairs based on the current policy's generation probability, effectively simulating on-policy learning. This approach improves upon Direct Preference Optimization (DPO), achieving up to 5.6% higher win rates against GPT-4-turbo on Alpaca Eval 2 and reaching a state-of-the-art 76.7% win rate with a Gemma-2-9b-it base model.

Key insights

  • Reweighting dispreferred (loser) outputs based on their likelihood under the current policy is more impactful for effective optimization than reweighting preferred outputs.
  • WPO's reweighting mechanism is universally applicable, consistently improving various existing preference optimization loss functions like IPO, SimPO, and KTO.
  • The hybrid RL setting, which combines both high-quality off-policy data and on-policy generated data, consistently yields the most robust performance for LLM alignment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.