Reproduce “HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuxuan Cai, Jiangning Zhang, Zhenye Gan, Qingdong He, Xiaobin Hu, Junwei Zhu, Yabiao Wang, Chengjie Wang and 4 more

The HumanVideo-MME benchmark was introduced to comprehensively evaluate Multimodal Large Language Models (MLLMs) on human-centric video understanding, assessing capabilities from basic perception to advanced cognitive reasoning. Evaluation of state-of-the-art MLLMs on this benchmark revealed strong performance in closed-form tasks but significant weaknesses in generative tasks and fine-grained perception.

Key insights

  • MLLMs exhibit a substantial performance disparity between closed-form (MC, TF) and generative (FIB, OEQ) tasks, indicating a reliance on superficial patterns or language priors in structured settings.
  • Current MLLMs demonstrate limitations in fine-grained perception tasks, such as face recognition, highlighting a bottleneck in capturing subtle identity-related visual cues.
  • The severe performance degradation in generative reasoning tasks, particularly for causal inference, suggests MLLMs struggle to produce coherent and contextually grounded explanations from scratch.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.