Reproduce “Hierarchical Reinforcement Learning with Timed Subgoals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nico Gürtler, Dieter Büchler, Georg Martius
HiTS (Hierarchical Reinforcement Learning with Timed Subgoals) introduces an HRL algorithm that uses timed subgoals to stabilize learning in dynamic environments. It achieves improved performance over existing methods on tasks requiring precise timing by ensuring the higher-level Semi-Markov Decision Process remains stationary.
Key insights
- Explicitly controlling the time interval for subgoal achievement via 'timed subgoals' is crucial for resolving the non-stationarity problem in HRL when operating in dynamic environments.
- Combining timed subgoals with Hindsight Action Relabeling renders the higher-level Semi-Markov Decision Process (SMDP) stationary, provided the environment's uncontrollable parts evolve independently during the specified time.
- The framework enables flexible temporal abstraction, allowing the higher level to adapt control granularity (e.g., smaller `∆t` for precision) for different task phases, implicitly guided by the Soft Actor-Critic's maximum entropy objective.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.