Reproduce “Meta Learning Shared Hierarchies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman
META LEARNING SHARED HIERARCHIES (MLSH) introduces an approach that combines meta-learning with hierarchical reinforcement learning to learn a set of reusable sub-policies across a distribution of tasks. The method allows agents to rapidly adapt to new tasks with higher sample efficiency and enables solving complex, sparse-reward environments by composing these pre-learned primitives.
Key insights
- Optimizing shared sub-policies (φ) to enable rapid adaptation and high performance for randomly initialized task-specific master policies (θ) is crucial for meta-learning a hierarchy.
- A 'warmup' period for the master policy (θ) ensures that updates to the shared primitives (φ) are based on a more effective, task-adapted master policy, preventing suboptimal primitive convergence.
- Temporal abstraction, where the master policy selects sub-policies at a slower rate, effectively reduces the horizon of the learning problem for the master policy, accelerating its adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.