Reproduce “Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge” on OpenResearch

About this paper

Fawaz Sammani, Nikos Deligiannis

This work introduces a training-free method to interpret CLIP's zero-shot image classification by quantifying the "mutual knowledge" between its vision and language encoders using fine-grained textual concepts. The study finds that a stronger shared knowledge, measured by Mutual Information (MI) dynamics, consistently correlates with higher zero-shot classification accuracy across diverse CLIP models.

Key insights

• The strength of shared knowledge, as measured by the AUC of MI dynamics, provides a robust, training-free metric that directly correlates with a CLIP model's zero-shot classification accuracy.
• CLIP's transformer-based (ViT) architectures consistently demonstrate stronger shared knowledge compared to CNN-based models, and larger models or higher-quality pretraining data generally lead to increased shared knowledge strength.
• The proposed method generates fine-grained, disentangled multimodal explanations (e.g., 'long feathered ears') that are both visually grounded and textually explicit, offering a deeper understanding of CLIP's reasoning than general attribution maps.

Summary by alphaXiv.