Reproduce “Concept-Aware Batch Sampling Improves Language-Image Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Adhiraj Ghosh, Vishaal Udandarao, Thao Nguyen, Matteo Farina, Mehdi Cherti, Jenia Jitsev, Sewoong Oh, Elisa Ricci and 2 more
CABS shapes vision-language training batches by annotating web images with concepts and selecting examples online from super-batches. DataConcept provides annotations for 128M image-text pairs. CABS-DM favors concept diversity for classification, while CABS-FM favors concept-dense images for retrieval. In CLIP ViT-B/32 tests, scores were 21.9 versus 17.3 on ImageNet and 16.4 versus 12.9 for retrieval. Evidence is limited to tested settings.
Key insights
- Batch composition can be optimized separately from the contrastive loss: concept metadata is used only to choose examples, leaving the CLIP or SigLIP objective unchanged.
- Different downstream tasks favor different data distributions—balanced concept coverage benefits classification, while visually complex, multi-object scenes benefit retrieval.
- Concept-aware recaptioning is complementary to sampling: longer captions grounded in detected visual concepts substantially improve performance even with ordinary IID training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.