Reproduce “Large-scale Reinforcement Learning for Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yinan Zhang, Eric Tzeng, Yilun Du, Dmitry Kislyuk
A large-scale reinforcement learning framework developed by researchers at Pinterest ATG and MIT CSAIL successfully fine-tuned text-to-image diffusion models. This framework substantially improved human preference alignment by achieving an 80.3% preference rate, significantly mitigated skintone bias to a statistical parity score of 0.453, and enhanced compositional accuracy, demonstrating effective multi-objective optimization.
Key insights
- Reinforcement learning, when scaled appropriately with techniques like per-batch reward normalization and clipped trust regions, can effectively fine-tune diffusion models on millions of prompts.
- Arbitrary and non-differentiable reward functions, including distribution-level rewards, can be successfully integrated to optimize for complex objectives like fairness and compositionality.
- Multi-objective joint training can mitigate the "alignment tax," allowing a single model to simultaneously improve on diverse objectives without significant performance degradation in any one area.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.