Reproduce “LLM-PySC2: Starcraft II learning environment for Large Language Models” on OpenResearch

About this paper

Zongyuan Li, Yanan Ni, Runnan Qi, Lumin Jiang, Chang Lu, Xiaojie Xu, Xiangbei Liu, Pengfei Li and 7 more

A new StarCraft II learning environment, LLM-PySC2, was developed to integrate large language models with the game's full complexity, offering complete action spaces and multi-modal observations for multi-agent collaboration. Experiments revealed LLMs struggle with domain knowledge, exhibit hallucinations, and face challenges in effective multi-agent coordination in complex real-time strategy scenarios, even with advanced models.

Key insights

• LLMs exhibit fundamental limitations in complex decision-making tasks, demonstrating a lack of specific game domain knowledge (e.g., Protoss shield mechanics) crucial for effective play.
• Several types of hallucinations were observed, including generating invalid actions, misidentifying units, and confusing coordinate systems, hindering reliable interaction with the environment.
• Simply scaling up LLM parameters or using purely reasoning-focused models did not consistently lead to improved performance in unfamiliar, complex environments, suggesting a need for specialized domain understanding.

Summary by alphaXiv.