Reproduce “Evaluating Large Language Models in Theory of Mind Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michal Kosinski
This research from Stanford University evaluates the performance of large language models on Theory of Mind tasks, finding that the most recent model, ChatGPT-4, achieved a 75% success rate on false-belief tasks, a capability level comparable to an average 6-year-old child.
Key insights
- Complex cognitive functions, such as Theory of Mind, can emerge spontaneously in Large Language Models (LLMs) as an unintended byproduct of their extensive language training.
- Applying rigorous psychological experimental methods is crucial for effectively understanding and evaluating the capabilities of complex, 'black-box' AI models like LLMs.
- LLMs are increasingly becoming implicit models of human psychological processes, suggesting they can serve as new tools for advancing psychological understanding of human cognition.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.