Reproduce “MSG-Chart: Multimodal Scene Graph for ChartQA” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yue Dai, Soyeon Caren Han, Wei Liu

MSG-Chart introduces a multimodal scene graph to explicitly represent structural and semantic relationships within charts, enhancing automatic Chart Question Answering. The framework consistently improved accuracy by up to 5.68% on the ChartQA dataset and up to 2.41 BLEU points on the OpenCQA dataset when integrated with existing vision-language models.

Key insights

  • Explicitly encoding structural and semantic relationships in charts through a multimodal scene graph leads to more robust chart comprehension than implicitly learning these complex patterns.
  • Designing textual graph edges based on predefined chart semantics and logical relationships (relation-based) is crucial for enhancing complex reasoning abilities, outperforming generic fully connected structures.
  • The MSG-Chart module functions effectively as a general plug-in, consistently enhancing various backbone architectures like UniChart and VL-T5 by providing explicit structural and semantic inductive biases.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.