Reproduce “WOLF: Werewolf-based Observations for LLM Deception and Falsehoods” on OpenResearch

About this paper

Mrinal Agarwal, Saad Rana, Theo Sundoro, Hermela Berhe, Spencer Kim, Vasu Sharma, Sean O'Brien, Kevin Zhu

Algoverse AI Research developed WOLF, a multi-agent social deduction benchmark using Werewolf, to assess Large Language Models' (LLMs) capacity for deception and its detection at the statement level. Experiments reveal that LLM Werewolves win 70% of games, and while suspicion for deceptive roles increases over time, LLM observers demonstrate moderate detection accuracy with 71-73% precision but 48-61% recall.

Key insights

• LLMs exhibit an asymmetry in social reasoning, demonstrating frequent self-reported deception (69.4% overall) but moderate detection capabilities (overall accuracy 0.52), resulting in Werewolves winning 70% of games.
• Longitudinal interactions allow LLMs to accumulate evidence, leading to a measurable increase in suspicion towards deceptive roles over time without significantly compounding errors against truthful agents.
• Subtle forms of deception, such as omission and misdirection, generate higher average suspicion and flagging rates than overt fabrication, indicating LLMs may be more vulnerable to nuanced deceit.

Summary by alphaXiv.