Reproduce “Theory of Mind for Multi-Agent Collaboration via Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, Katia Sycara
The paper evaluates Large Language Models (LLMs) in multi-agent collaboration and their Theory of Mind capabilities within a dynamic search and rescue text game. It demonstrates that GPT-4 agents, particularly when augmented with an explicit belief state representation, achieve high task completion and efficiency, comparable to state-of-the-art multi-agent reinforcement learning baselines, while also improving ToM inference.
Key insights
- LLMs, especially GPT-4, can exhibit sophisticated emergent collaborative behaviors such as task delegation, helping, and conflict resolution in zero-shot, decentralized multi-agent settings.
- Explicit belief state representation significantly improves LLM agent performance by mitigating systematic failures like long-horizon context neglect and hallucination, leading to more valid actions and higher efficiency.
- A dynamic evaluation framework revealed that LLMs possess strong introspective (97.2%) and first-order (80.1%) ToM capabilities, with second-order ToM (69.4%) remaining more challenging, especially with false beliefs or complex communication scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.