Reproduce “Exploring Visual Interpretability for Contrastive Language-Image Pre-training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, Xiaomeng Li
Research uncovers a universal tendency in Contrastive Language-Image Pre-training (CLIP) models to focus on background regions in visual explanations, attributing this to semantic shift in pooling modules. The proposed Explainable CLIP (ECLIP) framework corrects this, for instance improving mIoU by 22.78% to 27.26% and mSC by 41.69% to 55.35%, while retaining recognition performance.
Key insights
- Contrastive Language-Image Pre-training (CLIP) models universally exhibit an erroneous "background preference" in raw visual explanations, focusing on irrelevant regions.
- The root cause of this background preference is identified as "semantic shift" within the pooling module (attention pooling and average pooling), which disperses discriminative foreground features.
- Global Max Pooling (GMP) effectively counters this semantic shift, leading to accurate foreground localization in explanations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.