Reproduce “Sieve: Multimodal Dataset Pruning Using Image Captioning Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, Ari Morcos

Sieve introduces a multimodal dataset pruning technique that employs image captioning models to generate synthetic captions, which are then semantically compared against original alt-text using sentence transformers. This method, especially when combined with CLIPScore, improved average VLM performance by 2.6% on medium-scale and 1.7% on large-scale datasets across 38 downstream tasks.

Key insights

  • Image captioning models, particularly those trained on smaller, curated datasets, provide a valuable complementary signal for identifying true image-text alignment, reducing errors from noisy web-scraped alt-text.
  • Dedicated sentence transformers are more effective and efficient for semantic textual similarity comparison between generated captions and alt-text than general-purpose text encoders.
  • Fusing Sieve's caption-based alignment with CLIPScore's embedding similarity creates a synergistic pruning approach that mitigates individual weaknesses and enhances overall data quality for VLM training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.