Reproduce “Thinker: Learning to Plan and Act” on OpenResearch

About this paper

Stephen Chung, Ivan Anokhin, David Krueger

Thinker introduces an algorithm enabling reinforcement learning agents to learn planning autonomously within a learned world model, removing reliance on handcrafted planning methods. The approach achieves a 94.5% solving rate on Sokoban and a 261% median human-normalized score on Atari, demonstrating effective learned planning strategies.

Key insights

• An RL agent can be trained to autonomously learn sophisticated planning strategies by interacting with its own learned internal world model, eliminating the need for pre-defined planning algorithms.
• Framing planning within an augmented MDP allows standard model-free RL algorithms to effectively become model-based planners, increasing their generality and flexibility.
• The integration of a dual network for world model learning, coupled with a feature loss, improves the quality of task-relevant state predictions and overall model learning, leading to more robust planning.

Summary by alphaXiv.