Reproduce “Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers” on OpenResearch

About this paper

Jakub Grzywaczewski, Dawid Płudowski, Przemysław Biecek

This work explores the latent spaces of contrastively pre-trained Vision-Language Models (VLMs) by identifying a universal "noise subspace" through spectral decomposition of embedding covariance matrices. It demonstrates that pruning these non-semantic dimensions generally preserves or improves downstream task performance, such as increasing text-to-image alignment cosine similarity by up to +1.600 x 10^-2 for SigLIP models.

Key insights

• VLM latent spaces exhibit a universal structural property characterized by a consistent sharp drop in eigenvalues, indicating a shared "noise threshold" across diverse architectures and modalities.
• The identified noise dimensions are not class-specific but demonstrate strong subgroup invariance, with high alignment (over 90% for larger models) across diverse data subsets.
• A substantial fraction of VLM latent geometry is governed by shared, architecture-level noise, suggesting that dimensional collapse in VLMs results in dimensions storing non-semantic or irrelevant data.

Summary by alphaXiv.