Reproduce “Preference Optimization by Estimating the Ratio of the Data Distribution” on OpenResearch

About this paper

Yeongmin Kim, Heesun Bae, Byeonghu Na, Il-Chul Moon

Researchers from KAIST developed Bregman Preference Optimization (BPO), a framework that generalizes Direct Preference Optimization (DPO) for aligning large language models (LLMs) by estimating the ratio of data distribution using Bregman divergence. This method uniquely preserves DPO's theoretical optimality and practical simplicity while empirically improving both generation fidelity and diversity, achieving a state-of-the-art 55.9% length-controlled win rate on AlpacaEval2 for Llama-3-8B-Instruct.

Key insights

• The likelihood ratio of the optimal DPO policy can be entirely expressed using only the reference model and observed preference data, enabling a simpler and more theoretically sound generalization without intermediate models.
• Bregman divergence offers a flexible mathematical framework to generalize DPO, encompassing DPO as a special case and guaranteeing convergence to the same optimal policy while maintaining tractability.
• Analyzing gradients revealed that the choice of Bregman function dictates the point-wise gradient magnitude, providing a mechanism for dynamically weighting samples during optimization to enhance stability and performance.

Summary by alphaXiv.