Reproduce “Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zirui Song, Yuan Huang, Junchang Liu, Haozhe Luo, Chenxi Wang, Lang Gao, Zixiang Xu, Mingfei Han and 2 more
Researchers from MBZUAI and Northeastern University introduce WereBench, a high-quality multimodal dataset of human Werewolf gameplay, and WereAlign, a novel strategy-alignment evaluation framework. Benchmarking state-of-the-art large language models reveals their significant shortcomings in complex strategic reasoning, especially in deception and counterfactual trade-offs, with top models achieving an average accuracy of 72% on speech evaluation tasks.
Key insights
- Current LLMs exhibit a significant gap in human-level strategic reasoning in complex social contexts, particularly in deception reasoning and counterfactual trade-offs.
- Weaker LLMs benefit from explicit rule reminders, while rewriting persuasive speeches into objective summaries generally enhances decision-making by mitigating linguistic influence.
- Models perform relatively well in roles requiring the integration of dispersed public information (e.g., Witch role) and in generating persuasive statements, but struggle with direct verification and complex causal reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.