Reproduce “Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search” on OpenResearch

About this paper

Jonathan Light, Min Cai, Weiqin Chen, Guanzhi Wang, Xiusi Chen, Wei Cheng, Yisong Yue, Ziniu Hu

STRATEGIST introduces a bi-level framework that combines large language models (LLMs) for high-level strategic abstraction with Monte Carlo Tree Search (MCTS) for low-level policy refinement, enabling self-improvement in complex multi-agent adversarial games. The approach achieves human-comparable performance in Avalon and surpasses traditional reinforcement learning methods with significantly fewer training steps.

Key insights

• Combining LLMs' abstract reasoning with MCTS's precise look-ahead planning effectively overcomes the limitations of pure LLM agents in complex strategic foresight and consistent execution.
• Representing high-level strategies as human-readable text (e.g., Python code, natural language guides) significantly enhances interpretability and allows for modular, trackable improvements.
• Population-based self-play simulation provides superior, high-quality feedback for strategy refinement compared to single LLM critics or fixed opponent policies, fostering robustness.

Summary by alphaXiv.