Reproduce “Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qi Qian, Yuanhong Xu, Juhua Hu
Intra-Modal Proxy Learning (InMaP) enhances zero-shot visual categorization by learning class proxies directly in the vision embedding space from unlabeled target data. This method addresses the modality gap in models like CLIP, achieving 80.21% accuracy on ImageNet with ViT-L/14@336 and an average of 71.40% across 13 diverse downstream tasks with ViT-B/16.
Key insights
- The contrastive loss in CLIP, particularly with a small temperature parameter, preserves a significant "modality gap" between text and vision embedding spaces, even when optimizing for ranking.
- Optimal class proxies for visual classification tasks intrinsically reside within the vision space, making text-based proxies inherently sub-optimal due to this distributional mismatch.
- Learning vision-specific proxies by mimicking text-derived probability distributions using unlabeled target data provides a computationally efficient and effective way to mitigate the modality gap.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.