Reproduce “A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Karahan Sarıtaş, Kıvanç Tezören, Yavuz Durmazkeser
The authors review 58 papers on theory of mind in large language models, including 49 evaluating models. They organize benchmarks by mental-state abilities, task formats, and situatedness. Only 17 of the 49 papers compare models with people on the same task. The review finds emerging competence, but models underperform humans; it does not identify why.
Key insights
- LLMs show emerging competence in Theory of Mind (ToM) tasks, but persistent gaps remain in emulating full human cognitive abilities.
- GPT-based models are predominantly used in ToM research, and evaluations heavily rely on text-only modalities over multimodal approaches.
- LLM performance on ToM tasks is significantly influenced by task structure and prompting techniques, with issues like hallucinations and biases frequently occurring.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.