Reproduce “CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views” on OpenResearch

About this paper

Alexey Gavryushin, Dingxi Zhang, Zhao Huang, Alexandros Delitzas, Jiaqi Chen, Ben Ellis, Cedric Zöllner, Manthan Patel and 3 more

Researchers at ETH Zurich developed CoMind, a multimodal dataset featuring synchronized egocentric and exocentric video, gaze tracking, audio, and 3D environment data, to facilitate the study of Theory of Mind in human-human collaboration. Initial benchmarks using state-of-the-art vision-language models on tasks like joint attention and handover prediction indicate that domain-specific fine-tuning substantially improves social perception and proactive assistance capabilities.

Key insights

• Current state-of-the-art Vision-Language Models (VLMs) demonstrate significant deficiencies in zero-shot performance for real-world social perception tasks, particularly in spatial grounding.
• Fine-tuning open-source VLMs like Qwen3-VL on the CoMind dataset leads to substantial performance improvements across all tasks, highlighting the necessity of domain-specific training for complex social reasoning.
• While VLMs show some capability in semantic inference, they notably lack embodied visual grounding required for precise spatial localization and understanding of temporal action dependencies triggered by social cues in collaborative settings.

Summary by alphaXiv.