Reproduce “Stochastic Neural Networks for Hierarchical Reinforcement Learning” on OpenResearch

About this paper

Carlos Florensa, Yan Duan, Pieter Abbeel

A framework from UC Berkeley and OpenAI combines hierarchical reinforcement learning with stochastic neural networks to learn diverse, reusable skills, enabling efficient problem-solving in sparse reward and long-horizon environments. This approach significantly outperforms traditional baselines in continuous control tasks, demonstrating faster learning and enhanced spatial exploration.

Key insights

• Pre-learning a diverse set of fundamental "agent-space" skills in a simple environment can effectively simplify complex downstream tasks by abstracting away low-level actions.
• Stochastic Neural Networks, particularly with bilinear integration, can efficiently represent and learn multi-modal policies, allowing a single network to encapsulate varied skills.
• Explicitly maximizing mutual information between latent skill codes and agent state features is crucial for ensuring the learned skills are distinct and interpretable, preventing mode collapse.
• Hierarchical control over learned skills dramatically reduces the effective task horizon and provides structured exploration, leading to faster learning and improved performance in sparse-reward settings.

Summary by alphaXiv.