Reproduce “Diffusion Alignment as Variational Expectation-Maximization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaewoo Lee, Minsu Kim, Sanghyeok Choi, Inhyuck Song, Sujin Yun, Hyeongyu Kang, Woocheol Shin, Taeyoung Yun and 2 more
A framework called Diffusion Alignment as Variational Expectation-Maximization (DAV) fine-tunes diffusion models to optimize for external rewards while preserving sample diversity and naturalness. The method achieved an aesthetic reward of 8.04 in text-to-image synthesis and 7.71 predicted activity in DNA sequence design, consistently mitigating reward over-optimization and mode collapse.
Key insights
- Reformulating diffusion alignment as a variational Expectation-Maximization process can effectively mitigate both reward over-optimization and mode collapse.
- An iterative cycle of explicit test-time search for diverse, high-reward samples (E-step) and subsequent distillation into the model (M-step) is crucial for balanced optimization.
- The framework is versatile, operating on both continuous and discrete diffusion models and accommodating non-differentiable reward functions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.