Reproduce “Towards Understanding Graphical Perception in Large Multimodal Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kai Zhang, Jianwei Yang, Jeevana Priya Inala, Chandan Singh, Jianfeng Gao, Yu Su, Chenglong Wang

A new evaluation framework, grounded in graphical perception theory, systematically assesses Large Multimodal Models' (LMMs) ability to interpret charts, revealing that current LMMs heavily rely on text annotations and struggle to generalize across different visual encodings. This research identifies specific limitations in LMMs' understanding of visual data representations, such as poor generalization across chart types and dependence on numerical labels.

Key insights

  • LMMs exhibit limited generalization, showing significant performance drops when the same data is presented in different chart types (e.g., bar vs. scatter plot).
  • Models display a heavy reliance on numerical annotations within charts, with performance decreasing significantly (over 30 percentage points in some cases) when these labels are absent.
  • LMMs demonstrate weak compositional understanding of visual elements, often focusing on specific regions rather than interpreting underlying visual encoding principles.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.