Reproduce “MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings” on OpenResearch

About this paper

Ziyi Wang, Yuhang Wu, Dongxu Piao, Xingyu Liu, Tianhui Zhou, Miao Liu

Researchers from Tsinghua University and Duke University introduce MeetingToM, a benchmark designed to evaluate multimodal large language models' (MLLMs) Theory of Mind (ToM) capabilities in naturalistic multi-party meetings. This benchmark probes MLLMs' ability to infer latent social states, including distinguishing genuine consensus from pseudo-consensus, revealing that current MLLMs significantly lag behind human performance, particularly in multimodal integration and contextual reasoning.

Key insights

• MLLMs consistently underperform human annotators across all ToM tasks, with a substantial gap indicating persistent limitations in social intelligence for complex, naturalistic settings.
• Current MLLMs demonstrate challenges in effective multimodal integration; verbal cues alone sometimes yield better performance than combined multimodal input on higher-level tasks, suggesting difficulty fusing ambiguous visual and verbal evidence.
• Thinking prompts (e.g., Chain-of-Thought) and structured contextual priors (e.g., dialogue-act cues) offer inconsistent or even detrimental effects, highlighting MLLMs' struggle with calibrated contextual reasoning and appropriately weighing different forms of evidence.

Summary by alphaXiv.