Reproduce “EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models” on OpenResearch

About this paper

Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, Yang Liu

Researchers from Tsinghua University and 01.AI developed EgoThink, a comprehensive benchmark designed to evaluate Vision-Language Models' (VLMs) ability to reason and understand from a first-person perspective. The evaluation of 21 state-of-the-art VLMs using this benchmark revealed substantial limitations in their egocentric thinking capabilities, highlighting significant room for improvement across various cognitive dimensions.

Key insights

• All evaluated VLMs, including top models like GPT-4V, show considerable room for improvement in first-person perspective tasks, with average scores generally low across capabilities.
• Increasing the Large Language Model (LLM) parameter size generally improves performance, particularly when LLMs are not frozen during instruction tuning, suggesting the importance of LLM adaptability.
• Instruction tuning data, when not specifically aligned with first-person scenarios, yields only marginal improvements, indicating a potential mismatch between generic instruction tuning and egocentric understanding.

Summary by alphaXiv.