Reproduce “Sub-policy Adaptation for Hierarchical Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexander C. Li, Carlos Florensa, Ignasi Clavera, Pieter Abbeel

Researchers at UC Berkeley developed Hierarchical Proximal Policy Optimization (HiPPO), an algorithm for jointly training and adapting both high-level managers and low-level sub-policies in hierarchical reinforcement learning. HiPPO outperforms existing methods and flat PPO across various robotic locomotion tasks, demonstrating superior performance in learning from scratch and adapting pre-trained skills, particularly under dynamics perturbations.

Key insights

  • If skills are sufficiently distinct, the latent variable can be treated as an observed state for gradient estimation, simplifying the hierarchical policy gradient computation with minimal error.
  • Latent-dependent baselines can be used without introducing bias, effectively reducing variance in hierarchical policy gradient estimates and improving learning stability.
  • Varying the time-commitment of sub-policies during training acts as a powerful regularizer, leading to more robust and generalizable skills that adapt better to environmental changes and new tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.