Reproduce “On the role of planning in model-based deep reinforcement learning” on OpenResearch

About this paper

Jessica B. Hamrick, Abram L. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Buesing and 2 more

MuZero combines a learned model and search to improve learning targets and select training actions, with optional search at test time. Across eight tasks, combining search for learning and action selection during training reached a median normalized score of 90.3 across tasks, versus 46.7 for One-Step. Test-time search added less in most tasks; findings are limited to these MuZero settings.

Key insights

• Planning's most substantial contribution is during training, specifically for improving policy updates and generating a beneficial data distribution.
• Surprisingly, deep and complex search (e.g., deep MCTS, extensive UCT exploration) is often not necessary; shallow planning often suffices for strong performance in many environments.
• Generalization through planning, even with a perfect environment model, is heavily contingent on the generalization capabilities of the underlying learned policy prior and value function, not just the model's accuracy.

Summary by alphaXiv.