Reproduce “Why DPO is a Misspecified Estimator and How to Fix It” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Aditya Gopalan, Sayak Ray Chowdhury, Debangshu Banerjee

Researchers at the Indian Institute of Science, IIT Kanpur, and HP AI Research diagnosed DPO as a misspecified estimator for parametric language models and introduced AuxDPO, a new direct preference optimization algorithm. This method mitigates DPO's inherent limitations by preventing preference reversals and reward degradation, demonstrating consistent performance improvements across various LLM alignment tasks and datasets.

Key insights

  • DPO, when applied to parametric policies like LLMs, fundamentally solves a misspecified statistical estimation problem by performing a weighted KL-projection of the true reward onto a restricted, lower-dimensional manifold of implicit rewards.
  • This misspecification explains why DPO can lead to severe failure modes, such as policies that reverse human preferences or actively reduce expected reward, even when provided with infinite and perfectly clean preference data.
  • Analysis of RLHF reveals equivalence classes of reward functions that yield the same optimal policy, differing only by vectors in a specific null space; DPO's implicit reward is shown to be a minimum-norm representative within these classes, failing to capture the full true reward function.
  • The theoretical validity of local linear and quadratic approximations for both DPO's implicit reward manifold and the RLHF objective can be guaranteed by choosing a sufficiently large KL regularization parameter "beta".

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.