Reproduce “ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, Shafiq Joty

ChartGemma is a model designed for chart understanding and reasoning that generates its instruction-tuning data directly from chart images, overcoming the limitations of relying on underlying data tables. The model, built on the strongly aligned PaliGemma backbone, demonstrates improved performance across chart question answering, summarization, and fact-checking benchmarks, particularly on human-generated and real-world chart data.

Key insights

  • Synthesizing instruction-tuning data directly from chart images, rather than underlying data tables, leads to a model that better captures visual trends and generalizes more effectively to human-generated instructions and real-world charts.
  • Using a strongly aligned vision-language backbone model, such as PaliGemma, is crucial for superior chart understanding and reasoning, enabling the model to effectively interpret intricate relationships among diverse chart elements.
  • A single-stage fine-tuning approach on a well-aligned foundation model allows for rapid convergence and efficient training, producing a compact yet high-performing model for chart reasoning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.