Reproduce “Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models” on OpenResearch

About this paper

Sangwoong Yoon, Himchan Hwang, Dohyun Kwon, Yung-Kyun Noh, Frank C. Park

This work introduces Diffusion by Maximum Entropy Inverse Reinforcement Learning (DxMI), a framework that recasts diffusion model training as an inverse reinforcement learning problem to enable high-quality image generation with significantly fewer steps. The approach integrates a novel dynamic programming-based reinforcement learning algorithm and allows for stable, MCMC-free training of energy-based models, which are also shown to be effective for anomaly detection.

Key insights

• Reformulating diffusion model training as a maximum entropy inverse reinforcement learning problem allows for the discovery of efficient, short-run generation trajectories.
• Integrating the maximum entropy principle into the IRL framework stabilizes adversarial training and enables MCMC-free learning of Energy-Based Models (EBMs) by generating necessary negative samples.
• The Diffusion by Dynamic Programming (DxDP) algorithm provides an efficient, value-function based method to optimize discrete-time diffusion models, circumventing the need for full back-propagation through many steps and complex marginal entropy estimation.

Summary by alphaXiv.