Reproduce “GameArena: Evaluating LLM Reasoning through Live Computer Games” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang, Ion Stoica, Haojian Jin, Hao Zhang
GameArena evaluates language-model reasoning through three rule-bound games: Akinator, Taboo, and Bluffing. It combines game outcomes with retrospective, per-turn candidate lists or truthfulness judgments, which serve as proxies rather than verified records of internal reasoning. In Akinator, Claude 3.5 Sonnet’s recall was 27.83%, versus GPT-4o’s 20.32%; this task-specific metric is not a general reasoning ranking.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.