Reproduce “Visual Reinforcement Learning with Imagined Goals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, Sergey Levine
The Reinforcement Learning with Imagined Goals (RIG) framework learns broadly applicable, goal-conditioned policies directly from raw image observations on real robots. It employs a Variational Autoencoder for representation, automated goal generation, and a latent-space reward function, achieving high sample efficiency and successfully completing complex visuomotor tasks within hours of real-world interaction.
Key insights
- Distances in a learned VAE latent space provide a more effective and stable reward signal for visual goal-conditioned tasks than pixel-wise differences, enabling efficient learning from raw image observations.
- Combining automatically generated imagined goals (sampled from a VAE prior) with experience replay-based goal relabeling (like HER) significantly boosts sample efficiency and skill diversity, crucial for learning on real robots.
- Continuously fine-tuning the VAE alongside RL training allows the visual representation to adapt to new states explored by the policy, leading to improved performance and robustness over static representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.