Reproduce “First return, then explore” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, Jeff Clune

This research introduces Go-Explore, a new reinforcement learning paradigm designed to solve hard exploration problems in sparse-reward environments by explicitly remembering promising states and reliably returning to them before exploring further. The method achieved superhuman performance on all previously unsolved Atari games, including setting a new world record on Montezuma's Revenge, and successfully solved a complex robotics manipulation task.

Key insights

  • Explicitly preventing 'detachment' (forgetting how to reach states) and 'derailment' (exploration hindering return) is key to solving hard exploration problems.
  • A modular 'remember, return, then explore' strategy combined with learning from demonstration can achieve robust performance even in environments with sparse and deceptive rewards.
  • The approach is generalizable, effective both with simulator state restoration and a learned goal-conditioned policy for returning to states.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.