Reproduce “SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonathan Roberts, Kai Han, Neil Houlsby, Samuel Albanie
This paper introduces SciFIBench, a novel multiple-choice benchmark designed to evaluate Large Multimodal Models (LMMs) on their ability to interpret complex scientific figures. The evaluation reveals that even frontier LMMs achieve moderate performance, with closed-source models significantly outperforming open-source counterparts, and highlights issues with instruction following and reasoning faithfulness.
Key insights
- SciFIBench is a highly challenging benchmark, with even top LMMs demonstrating substantial room for improvement in scientific figure interpretation.
- A significant performance gap exists between closed-source and open-source LMMs, with the former consistently achieving higher accuracy across tasks.
- LMMs exhibit varying levels of instruction following and faithfulness in reasoning, indicating a susceptibility to external cues and potential 'cheating' rather than true understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.