Reproduce “Latent Plans for Task-Agnostic Offline Reinforcement Learning” on OpenResearch

About this paper

Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, Wolfram Burgard

TACO-RL, a hierarchical self-supervised framework, enables robots to perform complex, long-horizon tasks from high-dimensional visual observations using only offline and unstructured "play" data. This approach demonstrates substantial performance improvements, often by an order of magnitude, over leading baselines in both simulated and real-world multi-task robot manipulation scenarios.

Key insights

• Combining imitation learning for short-horizon skill acquisition and offline reinforcement learning for long-horizon goal chaining effectively addresses the limitations of both methods individually.
• Learning a continuous latent space of behaviors (latent plans) from unstructured 'play data' provides a flexible and task-agnostic action space for a high-level policy.
• The strategic use of hindsight relabeling with both positive and carefully sampled 'negative' goals is crucial for robustly learning Q-values and distances to goals from sparse, offline data, especially when generalizing to unseen goal images.

Summary by alphaXiv.