Reproduce “Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models” on OpenResearch

About this paper

Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, Roy Ka-Wei Lee

Researchers developed Math-LLaVA, a multimodal large language model fine-tuned on the synthesized MathV360K dataset, which achieves 46.6% accuracy on MathVista, marking a 19-point increase over the LLaVA-1.5 base model and demonstrating performance comparable to GPT-4V.

Key insights

• High-quality and diverse multimodal datasets, particularly those leveraging rich visual information, are crucial for enhancing MLLM mathematical reasoning.
• Systematic data augmentation techniques, including generating complex, rephrased, and underspecified questions via advanced LLMs, can significantly expand and improve training datasets.
• Training MLLMs on a dataset with a structured distribution of image comprehension complexity, favoring more complex examples, yields better performance in multimodal mathematical reasoning.
• Specialized training on multimodal mathematical reasoning data can also enhance a model's generalizability across diverse multidisciplinary understanding and reasoning tasks.

Summary by alphaXiv.