Reproduce “GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer and 1 more
This paper presents GAMEBENCH, a new cross-domain benchmark for evaluating strategic reasoning in Large Language Model (LLM) agents using obscure multi-player games designed to test out-of-distribution capabilities. The evaluation demonstrates that current LLMs remain significantly below human performance in strategic games, though Chain-of-Thought prompting generally improves their abilities, particularly for GPT-4.
Key insights
- Large Language Models (LLMs) consistently trail human performance in strategic reasoning across diverse, obscure game environments.
- Chain-of-Thought (CoT) prompting generally enhances LLMs' strategic reasoning, particularly for GPT-4, by eliciting step-by-step reasoning.
- The state-of-the-art Reasoning Via Planning (RAP) technique unexpectedly underperformed CoT prompting, potentially due to compounding errors or insufficient search depth.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.