Reproduce “A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)” on OpenResearch

About this paper

Weijie Tu, Weijian Deng, Tom Gedeon

This research provides a comprehensive empirical analysis of Contrastive Language-Image Pre-training (CLIP) models, detailing their robustness to visual factor variations, out-of-distribution (OOD) detection capabilities, and predictive uncertainty estimation. The study reveals that training data distribution and fine-tuning procedures significantly impact these safety-critical properties, challenging previous generalized assumptions about CLIP's inherent calibration and highlighting specific strengths and weaknesses across diverse conditions.

Key insights

• CLIP's safety-related properties, such as robustness and calibration, are heavily influenced by the training data distribution and quantity, often more so than by architecture or model size.
• While CLIP models show superior robustness to some visual factors and good OOD detection, their performance on specific factors (e.g., pose) or calibration can be inconsistent and sometimes inferior to baselines without careful configuration.
• Fine-tuning strategies, particularly multi-stage fine-tuning, can significantly improve OOD detection and calibration in CLIP models, revealing a nuanced trade-off between classification accuracy and other safety objectives.

Summary by alphaXiv.