Reproduce “Goal-Conditioned Reinforcement Learning with Imagined Subgoals” on OpenResearch

About this paper

Elliot Chane-Sane, Cordelia Schmid, Ivan Laptev

Reinforcement learning with Imagined Subgoals (RIS) introduces a self-supervised method where a high-level policy predicts intermediate "imagined subgoals" to guide the training of a single goal-conditioned policy via KL-regularized policy iteration. This approach enabled agents to achieve high success rates on challenging long-horizon robotic tasks with improved sample efficiency, including vision-based manipulation, compared to existing baselines.

Key insights

• Temporally abstract subgoals, when integrated into policy learning via regularization, can significantly accelerate the acquisition of long-horizon behaviors without needing explicit execution at test time.
• A policy's own value function can serve as a robust and dynamic distance metric, enabling self-supervised identification of meaningful intermediate subgoals.
• A virtuous cycle of learning emerges where a stronger policy yields better subgoal predictions, which in turn improves the policy further, extending the agent's solvable task horizon.

Summary by alphaXiv.