Reproduce “ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models” on OpenResearch

About this paper

Ziyue Wang, Chi Chen, Fuwen Luo, Yurui Dong, Yuanchi Zhang, Yuzhuang Xu, Xiaolong Wang, Peng Li and 1 more

Tsinghua University researchers introduce ActiView, a new benchmark to evaluate the active perception ability of Multimodal Large Language Models (MLLMs), which involves dynamically zooming and shifting perceptual fields. The evaluation reveals that current MLLMs, including state-of-the-art proprietary models like GPT-4o, significantly lag human performance in these tasks, achieving an average of 66.40% accuracy compared to human 84.67%.

Key insights

• Current MLLMs demonstrate a substantial gap in active perception compared to humans, suggesting a fundamental limitation in their ability to dynamically seek and integrate visual information.
• Models perform better on zooming tasks than on shifting tasks, indicating a struggle to form holistic understandings when information is distributed across sequentially perceived, constrained views.
• Multi-image MLLMs generally outperform single-image models, highlighting the architectural advantage of processing and reasoning across multiple visual inputs simultaneously for active perception.

Summary by alphaXiv.