Reproduce “Graph-Based Multimodal Contrastive Learning for Chart Question Answering” on OpenResearch

About this paper

Yue Dai, Soyeon Caren Han, Wei Liu

The research from Dai et al. introduces a graph-based multimodal contrastive learning framework to enhance Chart Question Answering by explicitly modeling relationships between chart components, leading to improved performance on the ChartQA and OpenCQA datasets. It also examines the effectiveness of Chain-of-Thought prompting for Multimodal Large Language Models in zero-shot chart understanding, noting inconsistent improvements and limitations with diverse chart types.

Key insights

• Explicitly modeling chart component relationships through multimodal scene graphs effectively retains and leverages object-level information often lost in patch-based vision-language models.
• Inter-modality graph contrastive learning, aligning visual and textual representations of chart objects, is more effective than intra-modality approaches for multimodal feature fusion in ChartQA.
• Chain-of-Thought prompting for MLLMs does not consistently improve zero-shot ChartQA performance, particularly with diverse chart types, and MLLMs show limitations in generating accurate long explanations.

Summary by alphaXiv.