Reproduce “Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhui Zhang, Elaine Sui, Serena Yeung-Levy
Researchers at Stanford University developed C3 (Connect, Collapse, Corrupt), a method that provides a theoretical explanation for the "modality gap" in multi-modal contrastive learning and significantly improves cross-modal task performance using only uni-modal data. C3 achieves state-of-the-art zero-shot results in image captioning, with a CIDEr score of 93.3, and enhances text-to-image generation quality.
Key insights
- The "modality gap" is a constant vector orthogonal to the effective embedding span, caused by dimensional collapse during initialization and preserved by contrastive optimization.
- Alignment noise arises from the "stable region" of the contrastive loss, where optimal paired embeddings can exist within a small, random Gaussian distribution.
- By addressing these precise geometric properties, it is possible to significantly enhance the interchangeability of embeddings across modalities for uni-modal training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.