Reproduce “MULTI: Multimodal Understanding Leaderboard with Text and Images” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zichen Zhu, Yang Xu, Lu Chen, Jingkai Yang, Yichuan Ma, Yiming Sun, Hailin Wen, Jiaqi Liu and 6 more
A new Chinese multimodal benchmark, MULTI, evaluates Multimodal Large Language Models (MLLMs) using authentic exam questions from Chinese scientific and educational contexts. It demonstrates a substantial performance gap, with the leading MLLM achieving 76.9% accuracy against human experts' 86.1%, highlighting current limitations in complex multimodal understanding.
Key insights
- Despite advancements, a substantial performance gap persists between even the most capable MLLMs and human experts on complex multimodal understanding and reasoning.
- MLLMs exhibit particular weaknesses in handling questions with multiple images, performing multi-step reasoning, and generating accurate open-ended responses.
- Effective integration of external knowledge for in-context learning is sensitive to context window size, while visual information and Chain-of-Thought prompting generally improve performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.