Reproduce “A Dense Reward View on Aligning Text-to-Image Diffusion with Preference” on OpenResearch

About this paper

Shentao Yang, Tianqi Chen, Mingyuan Zhou

Researchers at The University of Texas at Austin introduced a dense reward perspective with temporal discounting for aligning text-to-image diffusion models with human preferences, adapting a DPO-style objective. Their method achieved superior ImageReward, HPSv2, and Aesthetic scores, with human evaluations showing significantly higher win rates against baselines (e.g., 76.8% vs SD1.5, 65.1% vs trajectory-level DPO), and generated more coherent image structures earlier in the diffusion process.

Key insights

• Adopting a dense reward perspective with temporal discounting for T2I preference alignment significantly improves learning efficiency and stability compared to sparse, trajectory-level reward approaches.
• Emphasizing rewards from the initial steps of the diffusion process (via $\gamma < 1$) helps T2I models establish core image structures and outlines much earlier, facilitating better overall image quality.
• The derived DPO-style objective successfully bridges advanced reinforcement learning concepts with diffusion models, offering a principled way to align generative processes with sequential characteristics without an explicit reward model.

Summary by alphaXiv.