Reproduce “Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan and 1 more
Meta AI researchers introduce Distilled and Hard-negative Training (DiHT), a pre-training pipeline designed to enhance dual-encoder vision-language models by systematically improving data quality, knowledge transfer, and contrastive learning. DiHT models consistently outperformed CLIP baselines across 20 out of 29 zero-shot tasks while utilizing significantly smaller datasets, for instance, a DiHT-B/16 model on 438M samples achieved performance superior to CLIP-B/16 trained on 2B samples.
Key insights
- Systematically improving data quality through filtering is crucial for effective vision-language pre-training, enabling better performance with less data.
- Strong unimodal knowledge can be transferred to multimodal models efficiently without direct fine-tuning of the image encoder or significant computational overhead, preserving multimodal alignment.
- Re-weighting negative samples in contrastive learning to prioritize 'harder' negatives significantly enhances the discriminative power of learned embeddings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.