Reproduce “State-Conditioned Adversarial Subgoal Generation” on OpenResearch

About this paper

Vivienne Huiling Wang, Joni Pajarinen, Tinghuai Wang, Joni-Kristian Kämäräinen

State-Conditioned Adversarial Subgoal Generation (SAGA) addresses non-stationarity in Hierarchical Reinforcement Learning by using an adversarial mechanism to train a high-level policy to generate subgoals compatible with the low-level policy's current capabilities. This approach yields superior sample efficiency and asymptotic performance across a range of complex continuous control tasks, outperforming state-of-the-art hierarchical and flat RL baselines.

Key insights

• Adversarial training directly mitigates non-stationarity in off-policy HRL by forcing the high-level policy to generate subgoals that are compatible with the low-level policy's current capabilities.
• Conditioning the discriminator on the current state is crucial for accurately evaluating the compatibility and reachability of generated subgoals within the environmental context.
• The adversarial framework encourages the high-level policy to develop more effective long-term plans by producing semantically meaningful and reachable intermediate objectives.

Summary by alphaXiv.