Reproduce “Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models” on OpenResearch

About this paper

Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Juan Lao Tebar, Wenze Hu and 4 more

Apple AI/ML introduces a controllable captioning pipeline and demonstrates that a hybrid approach combining synthetic captions and web-crawled AltText generally outperforms using either data source alone for pre-training multimodal foundation models. The research highlights model-specific preferences, with CLIP benefiting from shorter captions and a 40-50% synthetic mix, while MLLMs prefer descriptive captions for improved performance in pre-training and supervised fine-tuning.

Key insights

• A hybrid approach leveraging both synthetic captions for image-text alignment and AltText for broader data diversity typically yields superior performance compared to using either alone.
• Different multimodal foundation models exhibit distinct preferences for caption characteristics: CLIP generally performs better with shorter synthetic captions, while MLLMs benefit from more descriptive and detailed captions.
• There is a trade-off between caption richness (information density) and factual accuracy (less hallucination), suggesting that optimal caption formats are task- and model-specific.

Summary by alphaXiv.