Reproduce “Understanding Contrastive Learning Requires Incorporating Inductive Biases” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy

The authors study why contrastive pretraining loss may not predict downstream classification quality. Their theory and experiments show that representation classes and training procedures shape transfer. In a CIFAR-10 hash-pixel test, ViT and MLP-Mixer achieve lower loss but near-random accuracy, while ResNet-18 retains good accuracy. The guarantees cover stated assumptions and linear representations, not deep networks generally.

Key insights

  • Downstream performance in contrastive learning is highly sensitive to the chosen function class (architecture) and training algorithm, even when contrastive loss values are similar.
  • Contrastive learning can successfully learn useful representations even when augmentations are explicitly designed to be disjoint, lacking shared information.
  • The relationship between contrastive loss and downstream performance can be non-monotonic, with continued minimization of contrastive loss sometimes leading to decreased downstream accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.