Reproduce “How Well Can Vison-Language Models Understand Humans' Intention? An Open-ended Theory of Mind Question Evaluation Benchmark” on OpenResearch

About this paper

Ximing Wen, Mallika Mainali, Anik Sen

Researchers at Drexel University evaluated Vision-Language Models' (VLMs) ability to understand human intentions using an open-ended Theory of Mind (ToM) question framework and a new 30-image benchmark. The study found that larger models like GPT-4 and GPT-4o-mini achieved high accuracy (27-28 out of 30) in inferring intentions, identifying visual cues, and making future predictions, significantly outperforming smaller models like LLaVA-7B, though all models struggled with nuanced contextual understanding.

Key insights

• Larger, more advanced Vision-Language Models (e.g., GPT-4, GPT-4o-mini) consistently demonstrate stronger Theory of Mind capabilities, inferring intentions and making future predictions more effectively than smaller models.
• All evaluated VLMs exhibit limitations in understanding nuanced contextual information and complex human intentions, such as distinguishing cheating behavior or correctly interpreting specific professional or religious attire.
• Smaller models can sometimes infer correct intentions despite reporting partially inaccurate visual cues, suggesting they may rely on broader contextual understanding rather than precise visual detail in certain instances.

Summary by alphaXiv.