Reproduce “Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text” on OpenResearch

About this paper

Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt and 2 more

This work introduces Multimodal C4 (`mmc4`), an open, billion-scale corpus comprising 101.2 million documents with 571 million images interleaved within 43 billion English text tokens. The dataset, created using a novel image-sentence alignment method, provides a crucial resource for training advanced vision-language models, demonstrated by a 20-30 CIDEr point improvement in few-shot image captioning for an OpenFlamingo model.

Key insights

• Large-scale, publicly available interleaved image-text data is essential for advancing open research in multimodal AI and enabling in-context learning capabilities.
• Zero-shot vision-language models can be effectively leveraged with bipartite linear assignment for high-quality, large-scale intra-document image-text alignment.
• Pretraining models on interleaved sequences like those in `mmc4` is crucial for developing robust multimodal in-context learning abilities and leads to improved generalization.

Summary by alphaXiv.