Reproduce “Robust Preference Optimization through Reward Model Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, Jonathan Berant
Google DeepMind and Google Research developed a reward model distillation framework, including pessimistic ensemble DPO (e-DPO), that enhances the robustness of language model alignment by mitigating Direct Preference Optimization's (DPO) susceptibility to spurious correlations and implicit reward magnitude issues, yielding improved performance in both biased and unbiased scenarios.
Key insights
- Theoretical analysis confirmed DPO's degenerate tendencies, showing its global minimizers can lead to infinite implicit rewards and shifting probability mass to unobserved outputs, particularly with disjoint preference pairs.
- Reward model distillation effectively reintroduces regularization, preventing the infinite reward magnitudes and likelihood collapse observed in DPO while maintaining an offline, DPO-like training framework.
- Pessimistic ensemble optimization (e-DPO) successfully mitigates the impact of spurious correlations in training data by allowing the policy to distill from a diverse set of reward signals, often implicitly matching behavior to ensemble members with opposing biases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.