Reproduce “Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zixiang Chen, Yihe Deng, Yuanzhi Li, Quanquan Gu

The method adds a positive-pair similarity term to CLIP’s contrastive objective, addressing cases where an unpaired image and caption may share a feature. It retains the batchwise contrastive loss and rewards each observed pair, without treating unknown pairings as reliable negatives. In a CC3M-pretrained ResNet-50/DistilBERT setup, it improves zero-shot accuracy on five of six benchmarks, but not DTD.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.