Reproduce “CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement” on OpenResearch

About this paper

Mohammadreza Salehi, Mehrdad Farajtabar, Maxwell Horton, Fartash Faghri, Hadi Pouransari, Raviteja Vemulapalli, Oncel Tuzel, Ali Farhadi and 2 more

Researchers from Apple, Inc. developed CLIPTeX, a method enhancing Contrastive Language-Image Pre-training (CLIP) models by integrating pseudo-supervision from publicly available, task-specific expert models. This approach improved visual representations for dense prediction tasks, achieving approximately 10-16.3% higher mIoU on PASCAL VOC segmentation benchmarks while preserving CLIP's zero-shot generalization capabilities.

Key insights

• Expert knowledge from diverse, publicly available task-specific models can be effectively transferred to a general-purpose vision-language foundation model through pseudo-supervision on web-scale data.
• Multi-objective training with pseudo-labels can enhance CLIP's visual representations for localization and dense prediction tasks without compromising its pre-existing zero-shot classification and cross-modal retrieval capabilities.
• Lightweight task heads and a multi-scale module are sufficient for efficient knowledge transfer, leading to improved performance across various downstream tasks even for classes not explicitly present in the expert's training data.

Summary by alphaXiv.