Reproduce “Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards” on OpenResearch

About this paper

Siyuan Li, Rui Wang, Minxue Tang, Chongjie Zhang

HAAR introduces a hierarchical reinforcement learning framework that uses high-level advantage functions as auxiliary rewards, enabling low-level skill adaptation and concurrent policy optimization in sparse reward environments. It demonstrated superior performance against state-of-the-art baselines and exhibited policy transferability across various continuous control tasks.

Key insights

• High-level advantage functions can serve as generic, task-agnostic auxiliary rewards, effectively propagating sparse environment signals to guide low-level skill adaptation without requiring domain-specific knowledge.
• The proposed concurrent optimization of both hierarchical levels, supported by theoretical guarantees for monotonic improvement, is an effective and efficient strategy for learning in sparse reward environments.
• Careful design of observation spaces, utilizing ego-centric observations for low-level skills and range sensors without absolute coordinates for high-level decisions, promotes the learning of generalizable and transferable policies.

Summary by alphaXiv.