Reproduce “$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, Minkai Xu
f-PO recasts offline language-model preference optimization as distribution matching, allowing researchers to choose among f-divergences rather than defaulting to one loss. It matches policy to a reward-defined target, and its benchmark version uses a SimPO-style approximation. For Llama-3-8B-Instruct, it reports a 45.3% length-controlled win rate on AlpacaEval 2 versus SimPO’s 44.7%, edging ahead on that metric but not every evaluation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.