Reproduce “NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang and 2 more
Researchers from The Hong Kong University of Science and Technology and Tencent AI Lab introduced NegotiationToM, a benchmark for evaluating multi-dimensional Theory of Mind (ToM) in large language models (LLMs) using real-world human-to-human negotiation dialogues. The benchmark revealed a substantial performance gap between state-of-the-art LLMs and human evaluators in inferring desires, beliefs, and intentions, highlighting current models' limitations in dynamic social reasoning.
Key insights
- LLMs struggle significantly with coherent and simultaneous understanding of multi-faceted mental states, demonstrated by their very low "All" scores (e.g., best LLM at 3.68% versus human 43.78%).
- Current LLMs have limited ability to consistently track and dynamically update mental states (desires, beliefs, intentions) throughout an entire negotiation conversation, indicating a lack of robust long-term social reasoning.
- Explicitly providing desire and belief information empirically improves LLM performance in related downstream tasks, such as negotiation strategy prediction, validating the utility of these mental states as valuable intermediate representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.