Reproduce “Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization” on OpenResearch

About this paper

Ziqi Wang, Jiashun Liu, Ling Pan

The DrAC algorithm enables deep reinforcement learning agents to acquire intractable multimodal policies through a reparameterization trick and a novel distance-based diversity regularization. This approach yields agents with enhanced few-shot robustness in dynamic environments and superior quality-diversity trade-offs in generative reinforcement learning tasks, while maintaining computational efficiency with amortized actors.

Key insights

• The reparameterization trick can be generalized to a broad class of intractable stochastic policies, allowing for efficient online optimization of highly expressive multimodal actor architectures.
• Maximizing the geometric mean of pairwise distances between sampled actions (via a log-distance objective) is an effective and principled way to induce meaningful multimodality in policies, outperforming traditional entropy regularization.
• Amortized actors, when integrated with a proper diversity regularization scheme, offer a compelling balance of expressivity, computational efficiency, and multimodal capability, demonstrating advantages over more complex diffusion actors in several diversity-critical tasks.

Summary by alphaXiv.