Reproduce “Disentangling Optimization Scale from Preference Scale in DPO” on OpenResearch

About this paper

Ivan Kruzhilov

The research uncovers that the `β` parameter in Direct Preference Optimization (DPO) simultaneously governs the preference-noise scale and directly rescales optimization dynamics, leading to non-monotone policy deviation and incomparable loss values. A normalized DPO objective is introduced, which disentangles these roles, allowing `β` to solely control the preference scale and enabling more predictable training behavior with interpretable loss metrics.

Key insights

• The `β` parameter in DPO significantly impacts optimization dynamics by directly scaling gradients, leading to a 'dead zone' at small `β` and a non-monotone policy deviation curve.
• Standard DPO loss values are not comparable across different `β` settings because the loss depends only on the product `β∆`, masking true policy displacement and learning progress.
• Decoupling `β`'s gradient-scaling role from its preference-noise scale role is crucial for predictable optimization and interpretable training metrics.

Summary by alphaXiv.