Reproduce “Diffusion-Reward Adversarial Imitation Learning” on OpenResearch

About this paper

Chun-Mao Lai, Hsiang-Chun Wang, Ping-Chun Hsieh, Yu-Chiang Frank Wang, Min-Hung Chen, Shao-Hua Sun

A novel adversarial imitation learning framework, Diffusion-Reward Adversarial Imitation Learning (DRAIL), integrates a conditional diffusion model into the discriminator of Generative Adversarial Imitation Learning (GAIL). This approach overcomes the inherent training instability of GAIL by generating smoother and more robust reward signals, leading to improved policy learning across various continuous control tasks.

Key insights

• Conditional diffusion models can serve as robust discriminators in adversarial imitation learning by explicitly comparing the fit of state-action pairs to expert versus agent distributions.
• Diffusion loss, serving as an upper bound for negative log-likelihood, can be efficiently approximated with a single denoising step, making it computationally feasible for online reward generation.
• Utilizing a diffusion-based discriminator leads to the generation of smoother and more robust reward functions, which are crucial for stable policy learning and generalization.

Summary by alphaXiv.