Reproduce “II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, Zekun Wang, Yuelin Bai, Qixuan Zhao and 18 more
Researchers from a large consortium led by Shenzhen Institutes of Advanced Technology introduced II-Bench, a benchmark designed to evaluate multimodal large language models' (MLLMs) capacity for understanding complex image implications. Their evaluation of 20 MLLMs revealed a significant performance gap against human understanding, with the best model achieving 74.8% accuracy compared to human average of 90.3%, particularly struggling with abstract concepts and exhibiting a 'positive emotion bias'.
Key insights
- There is a substantial performance disparity between MLLMs and human cognition in interpreting complex image implications, indicating significant limitations in current AI's higher-order perceptual abilities.
- MLLMs exhibit a 'positive emotion bias,' performing better on images with positive emotional tones and struggling more with negative ones, suggesting a deficiency in emotional understanding.
- Contrary to expectations, Chain-of-Thought and few-shot prompting often did not improve or sometimes even degraded MLLM performance on this task, potentially due to limitations in instruction following or context handling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.