Reproduce “Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games” on OpenResearch

About this paper

Nathan Herr, Fernando Acero, Roberta Raileanu, María Pérez-Ortiz, Zhibin Li

Researchers at University College London, with affiliation from FAIR at Meta, systematically investigated how Large Language Models (LLMs) exhibit and are affected by systematic biases in two-player non-zero-sum games. The study demonstrated that prompt-dependent biases significantly influence LLM strategic choices, leading to performance drops ranging from 16% to 34% when biases were misaligned with optimal actions.

Key insights

• Large Language Models (LLMs) are systematically influenced by prompt-dependent biases (positional, payoff, behavioral) when making strategic decisions in two-player non-zero-sum games.
• The observed 'cooperative' or 'rational' behavior in LLMs might not stem from true strategic reasoning but rather from a 'skewed attention' to specific outcomes or prompt elements.
• Chain-of-Thought (CoT) prompting can reduce the impact of biases for some models but can also exacerbate them for others, indicating it is not a universal solution.

Summary by alphaXiv.