Reproduce “SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis” on OpenResearch

About this paper

Ehud Gordon, Meir Yossef Levi, Guy Gilboa

SCoCCA introduces a framework for concept-based explainability in multimodal vision-language models, leveraging Canonical Correlation Analysis (CCA) and sparsity to generate disentangled and semantically pure concepts. It addresses the modality gap in models like CLIP by providing a training-free mechanism for cross-modal alignment, achieving high reconstruction fidelity and enabling precise concept editing.

Key insights

• Optimizing Canonical Correlation Analysis (CCA) implicitly enhances the alignment component of the InfoNCE loss when applied to whitened embeddings, offering a training-free mechanism for cross-modal alignment.
• Explicit sparsity regularization is crucial for obtaining disentangled and discriminative concepts in multimodal embeddings, improving interpretability and controllability.
• The proposed framework effectively bridges the modality gap in vision-language models like CLIP by identifying directions in the embedding space that are maximally aligned between modalities.

Summary by alphaXiv.