Reproduce “Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, Vasant G Honavar

Cal-DPO introduces a calibrated direct preference optimization method that addresses the scale invariance issue in existing LLM alignment techniques by explicitly guiding the absolute implicit rewards of chosen and rejected responses. This approach leads to consistent performance gains across diverse reasoning, instruction-following, and human preference benchmarks.

Key insights

  • The scale invariance in DPO-like methods can lead to suboptimal alignment by allowing chosen response likelihoods to decrease, even if they remain relatively better.
  • Explicitly calibrating implicit rewards ensures chosen responses' likelihoods increase in absolute terms, promoting truly higher quality outputs.
  • The calibration enables "mode-seeking" behavior, concentrating probability mass on high-reward regions, which is a desirable property for LLM generation similar to RLHF.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.