Reproduce “Monte-Carlo Tree Search for Policy Optimization” on OpenResearch

About this paper

Xiaobai Ma, Katherine Driggs-Campbell, Zongzhang Zhang, Mykel J. Kochenderfer

Researchers from Stanford University, University of Illinois Urbana-Champaign, and Nanjing University developed Monte-Carlo Tree Search for Policy Optimization (MCTSPO), a novel method adapting MCTS to continuous policy spaces in deep reinforcement learning. MCTSPO consistently outperformed gradient-based and other gradient-free baselines across challenging continuous control tasks with sparse or deceptive reward functions, effectively mitigating local optima issues.

Key insights

• Formulating policy optimization as an MDP allows for a principled, tree-based approach like MCTS to effectively balance exploration and exploitation within continuous, high-dimensional policy parameter spaces.
• The MCTS UCB exploration mechanism, combined with its tree search structure, effectively mitigates local optima issues that frequently trap both gradient-based and simpler gradient-free DRL methods.
• Specific adaptations to MCTS, such as progressive widening and MaxUCT, are crucial for successfully applying tree search to the continuous, high-dimensional parameter spaces inherent in deep policy networks.

Summary by alphaXiv.