Reproduce “Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Max Lamparth, Anthony Corso, Jacob Ganz, Oriana Skylar Mastro, Jacquelyn Schneider, Harold Trinkunas
Researchers at Stanford University compared large language models to national security experts in a military wargame simulation, finding LLMs exhibited high-level behavioral agreement but also distinct strategic biases, inconsistent instruction following, and a complete inability to model individual player characteristics or realistic group deliberation. The study concludes that current LLMs introduce unpredictable risks for high-stakes military decision-making.
Key insights
- LLMs, while showing high-level agreement with human responses, exhibit systematic granular deviations in strategic preferences, such as specific action choices in crisis scenarios.
- The quality of LLM-simulated group dialog is poor, lacking genuine interaction and disagreement, and LLMs cannot account for individual player characteristics, even extreme ones like "pacifist" or "sociopath."
- LLM behavior is highly sensitive to prompt engineering, specific model choice, and the absence of explicit instructions, leading to unpredictable strategic tendencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.