Reproduce “DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy” on OpenResearch

About this paper

Kaixuan Xu, Jiajun Chai, Sicheng Li, Yuqian Fu, Yuanheng Zhu, Dongbin Zhao

Researchers at the University of Chinese Academy of Sciences developed DipLLM, an approach that fine-tunes Large Language Models to achieve superior strategic decision-making in the complex multi-player game of Diplomacy. It outperformed the state-of-the-art Cicero by 2.2% in sum-of-squares score, utilizing only 1.5% of the data required by previous methods.

Key insights

• Fine-tuning Large Language Models with a combination of autoregressive factorization and an equilibrium-driven learning objective enables human-level strategic decision-making in highly complex multi-agent games like Diplomacy.
• LLMs, when properly adapted, can achieve state-of-the-art performance in complex strategic domains with significantly higher data efficiency compared to traditional domain-specific reinforcement learning methods.
• Decomposing intractable joint action spaces into sequential sub-action spaces is a crucial technique for leveraging the generative capabilities of LLMs in environments with combinatorial explosion of actions.

Summary by alphaXiv.