Reproduce “Direct Alignment with Heterogeneous Preferences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ali Shirali, Arash Nasr-Esfahany, Abdullah Alomar, Parsa Mirtaheri, Rediet Abebe, Ariel Procaccia
The authors show that DPO trained on mixed preferences may rank answers by normalized Borda count rather than average reward, with rankings depending on sampled alternatives. They develop corrections using annotator information and prove that, under Bradley–Terry assumptions and restricted losses, consistent direct alignment discards disagreements. In a simulated Llama-3-8B test, agreement rises from 67.4% to 71.7%, with both loss and feedback protocol changing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.