Reproduce “CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingting Gao, Yu Wu, Long Chen
This paper introduces CoMM, a large-scale, high-quality interleaved image-text dataset designed to improve the coherence and consistency of multimodal large language models (MLLMs). The dataset, constructed through targeted sourcing and a multi-perspective filtering strategy, enables MLLMs to generate more logical narratives and instructions with better image-text alignment and visual continuity.
Key insights
- High-quality, coherent, and consistent training data is crucial for unlocking the full potential of multimodal large language models (MLLMs), shifting focus to a data-centric approach.
- A multi-perspective filtering strategy, combining LLM-based text coherence checks, CLIP-based image consistency metrics, and MLLM-based image-text alignment, is effective for curating high-fidelity interleaved multimodal data.
- Novel benchmark tasks and evaluation metrics, including Illustration Relevance Score (IRS) and GPT-4o-based coherence assessments, are essential for evaluating complex interleaved generation capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.