Reproduce “Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, Rob Fergus

A self-play scheme involving two agents, Alice and Bob, enables unsupervised reinforcement learning by allowing Alice to propose tasks for Bob, who attempts to solve them. This approach automatically generates a curriculum of increasingly complex tasks, leading to faster learning and improved sample efficiency for downstream sparse reward tasks across various environments.

Key insights

  • The asymmetric reward structure naturally forms an automatic curriculum, with Alice adapting task difficulty to Bob's current skill level.
  • In a tabular setting, the system theoretically converges to a state where Bob learns a "fast policy" capable of transitioning between any two states efficiently.
  • The self-play mechanism intrinsically encourages broad exploration and understanding of environment dynamics without relying on explicit novelty bonuses.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.