Reproduce “Toward a Holistic Evaluation of Robustness in CLIP Models” on OpenResearch

About this paper

Weijie Tu, Weijian Deng, Tom Gedeon

A comprehensive evaluation of Contrastive Language-Image Pre-training (CLIP) models analyzes their robustness across specific visual factors, out-of-distribution detection, confidence calibration, zero-shot retrieval, and 3D awareness, as well as their interaction within large multimodal models. This research from The Australian National University identifies crucial factors influencing CLIP's performance and highlights architectural and training-dependent trade-offs for future foundation model development.

Key insights

• No single training paradigm or model universally excels across all evaluation metrics, highlighting inherent trade-offs between different aspects of robustness, accuracy, and calibration.
• Training distribution and visual encoder architectural choices (e.g., CNN vs. ViT) profoundly influence specific robustness dimensions, such as visual factor robustness, OOD detection performance, and 3D awareness.
• Fine-tuning strategies can introduce trade-offs, often boosting accuracy while degrading calibration or shifting biases (e.g., from shape to texture), necessitating careful design to preserve desirable zero-shot properties and achieve robust outcomes.

Summary by alphaXiv.