Reproduce “Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals” on OpenResearch

About this paper

Vivienne Huiling Wang, Tinghuai Wang, Joni Pajarinen

HIDI introduces a hierarchical reinforcement learning approach that combines conditional diffusion models and Gaussian Processes to generate effective subgoals for complex, long-horizon tasks. This method consistently outperforms state-of-the-art baselines in learning stability, sample efficiency, and asymptotic performance across various continuous control environments, demonstrating up to 16% improvements from its key components.

Key insights

• The non-stationarity problem in HRL can be effectively mitigated by a high-level policy that adaptively generates subgoals aligned with the current low-level capabilities.
• Integrating conditional diffusion models with Gaussian Processes creates a synergy where diffusion models provide expressive subgoal distributions, and GPs offer crucial uncertainty quantification and regularization for robust and sample-efficient learning.
• Guiding subgoal generation with explicit uncertainty estimates from a Gaussian Process allows the diffusion model to focus learning on feasible regions, leading to more stable and reliable policy updates.

Summary by alphaXiv.