Reproduce “Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning” on OpenResearch

About this paper

Chuan Qin, Constantin Venhoff, Sonia Joseph, Fanyi Xiao, Stefan Scherer

SPARSE CLIP, developed by researchers at Meta and the University of Oxford, integrates non-negative constraints and dimension expansion directly into the CLIP training process to co-optimize interpretability and performance. This approach yields inherently sparse, multimodal representations that achieve competitive or superior zero-shot classification performance while enabling direct conceptual understanding and vision-based steering of VLMs.

Key insights

• Interpretability and performance in contrastive learning can be co-optimized through native sparse training, directly challenging the conventional trade-off between these two objectives.
• The combination of non-negative constraints and high-dimensional projection is critical for inducing beneficial sparsity and disentangling features while maintaining or improving downstream task performance.
• The proposed method leads to features that are genuinely multimodal, activating for both image and text inputs, which facilitates direct semantic concept alignment and a deeper understanding of cross-modal knowledge acquisition.

Summary by alphaXiv.