Reproduce “SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs” on OpenResearch

About this paper

Xin Su, Man Luo, Kris W Pan, Tien Pei Chou, Vasudev Lal, Phillip Howard

A large-scale synthetic multimodal dataset, SK-VQA, was developed to address data scarcity for training context-augmented Multimodal Large Language Models (MLLMs). The dataset, containing over 2 million QA pairs, was generated using GPT-4 to simultaneously produce context documents and question-answer pairs for diverse images, demonstrating improved out-of-domain generalization for MLLMs trained on it and serving as a challenging benchmark for current models.

Key insights

• Synthetically generated data, specifically from powerful models like GPT-4, can effectively train Multimodal Large Language Models (MLLMs) for context-augmented generation, even outperforming or matching training on real data from certain sources.
• The simultaneous generation of context documents and question-answer pairs by a large language model (LLM) significantly enhances the relevance and utility of the generated data for training context-aware MLLMs.
• SK-VQA serves as a challenging benchmark for current MLLMs, demonstrating that simply scaling model size does not proportionally improve performance on complex reasoning tasks requiring external knowledge.

Summary by alphaXiv.