Reproduce “HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks” on OpenResearch

About this paper

Ting Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding, Yaliang Li, Ying Shen

HUMANVBENCH introduces an automatically synthesized benchmark and evaluation framework for Multimodal Large Language Models (MLLMs), specifically targeting fine-grained human-centric video understanding. It reveals that current MLLMs exhibit a notable gap from human performance across tasks involving emotion perception, person recognition, behavior analysis, and speech-visual alignment, and significantly reduces manual annotation effort for benchmark creation.

Key insights

• Current MLLMs, including proprietary models, show a substantial performance gap from human-level understanding (human average 88.6%) in human-centric video tasks.
• Models struggle with subtle emotion perception, precise speech-visual alignment (e.g., lip-reading), and maintaining performance in high-density human scenes, often exhibiting errors related to temporal noise or insufficient spatial-temporal grounding.
• The automated benchmark generation process significantly reduces manual labor, with 72% of descriptive questions requiring no human revision, while creating diagnostically valuable distractors from model-induced errors.

Summary by alphaXiv.