Reproduce “Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhikai Li, Yue Zhao, Edward Zhongwei Zhang, Xuewen Liu, Jing Zhang, Qingyi Gu, Zhen Dong
ArenaPO introduces an offline preference optimization method for diffusion models that estimates fine-grained quality gaps from binary preference data, improving alignment with human preferences. This approach achieves superior performance across multiple T2I benchmarks like PickScore and HPS v2, surpassing prior DPO methods by leveraging nuanced feedback without an explicit reward model.
Key insights
- Binary human preference datasets can be reinterpreted to establish probabilistic capability distributions for source generative models, providing a richer context than simple pairwise labels.
- The absolute quality difference between a preferred and a rejected image can be inferred and quantified offline using latent-variable inference and properties of the truncated normal distribution, without needing an explicit reward model.
- Incorporating a continuous, instance-specific quality gap as a margin within the DPO loss function enables more precise preference alignment, improving upon the limitations of fixed or binary margins.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.