Reproduce “ChartM$^3$: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension” on OpenResearch

About this paper

Duo Xu, Hao Cheng, Xin Lin, Zhen Xie, Hao Wang

Alibaba Cloud Computing researchers developed ChartM3, a comprehensive dataset for chart comprehension built through a multi-stage, code-driven pipeline, which generates multi-dimensional and multi-step visual reasoning data. The dataset includes 62 chart subcategories and incorporates Chain-of-Thought reasoning, enabling MLLMs fine-tuned on ChartM3 to achieve at least 12% performance gains and enhanced generalization through Reinforcement Learning with Verifiable Reward (RLVR).

Key insights

• ChartM3 serves as an effective benchmark, revealing significant performance gaps among MLLMs (over 15%) on complex reasoning tasks, which existing simpler benchmarks often fail to differentiate.
• Supervised fine-tuning (SFT) with ChartM3's high-quality CoT data leads to substantial performance improvements (at least 12%) and enhanced out-of-domain generalization for MLLMs in chart comprehension.
• Reinforcement Learning with Verifiable Reward (RLVR), a novel application in this domain, further boosts MLLM generalization capabilities, particularly for reasoning-intensive tasks, complementing the strengths of SFT.

Summary by alphaXiv.