Reproduce “Text-To-Concept (and Back) via Cross-Model Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, Soheil Feizi
A method for aligning the representation space of any pre-trained vision model to CLIP's shared embedding space enables "text-to-concept" and "concept-to-text" functionalities using only a linear transformation. This approach allows existing uni-modal models to perform zero-shot classification and provides concept-based interpretability without additional training or concept supervision.
Key insights
- A simple linear transformation is sufficient to effectively align the representation spaces of diverse deep vision models, including those with different architectures, training objectives, and datasets.
- Deep models learn to organize information in surprisingly similar semantic ways, suggesting a universal structure in their representations despite disparate training.
- Existing uni-modal vision models possess a richer understanding of semantic concepts than their explicit training supervision suggests, enabling emergent zero-shot abilities when aligned with multi-modal models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.