Reproduce “ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mind” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaomeng Ma, Lingyu Gao, Qihui Xu
The paper introduces TOMCHALLENGES, a principle-guided dataset and evaluation framework designed with established psychological Theory of Mind (ToM) tests to rigorously assess ToM capabilities in large language models (LLMs). Evaluations using this framework revealed that current LLMs, including GPT-4, exhibit inconsistent and unreliable ToM performance across diverse tasks and question complexities.
Key insights
- Robust ToM assessment in LLMs requires principle-guided evaluation that precisely distinguishes true ToM (mentalizing, false-belief understanding) from general social inference.
- LLMs' performance on ToM tasks is highly sensitive to the specific task format, question complexity, and even subtle linguistic variations within narratives, indicating a lack of generalized ToM.
- LLM failures on ToM tasks are not solely due to a lack of reasoning but also include 'overly conservative' refusals to infer and 'hallucinations,' highlighting intrinsic limitations beyond cognitive ToM explanations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.