Reproduce “MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zifeng Zhu, Mengzhao Jia, Zhihan Zhang, Lang Li, Meng Jiang
MultiChartQA introduces a benchmark dataset to evaluate Vision-Language Models (MLLMs) on complex multi-chart comprehension and multi-hop reasoning tasks. Evaluations on 20 MLLMs using 500 real-world multi-chart sets and 2,000 annotated question-answer pairs revealed a substantial performance gap against human accuracy of 90.11%, with the leading model achieving 77.72%, particularly in sequential reasoning.
Key insights
- A significant performance gap exists between current MLLMs and human intelligence in multi-chart reasoning, with even top models substantially underperforming human experts.
- MLLMs face particular challenges with sequential reasoning tasks, which demand complex multi-hop information tracking across charts, and are susceptible to distraction from irrelevant visual information.
- Chain-of-Thought reasoning and explicit chart references generally improve MLLM performance, indicating that models benefit from structured guidance in complex multi-chart contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.