Reproduce “Preference Learning Algorithms Do Not Learn Preference Rankings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, Kyunghyun Cho
Research by Chen et al. reveals that state-of-the-art preference-tuned large language models, despite training with methods like DPO, exhibit surprisingly low ranking accuracies, typically under 60%. This work demonstrates a significant "alignment gap" between observed performance and theoretically achievable ideals, primarily because DPO struggles to correct initial ranking errors present in the reference model.
Key insights
- State-of-the-art preference-tuned LLMs exhibit surprisingly low ranking accuracies, frequently below 60%, even after extensive training.
- DPO primarily reinforces existing correct preferences and struggles to correct initial ranking errors present in the reference model.
- The reference model's initial likelihoods significantly condition the DPO optimization landscape, making it exponentially harder to correct larger initial ranking errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.