Reproduce “A Theory of Contrastive Learning with Natural Images” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Antonio Torralba, Yair Weiss

Antonio Torralba and Yair Weiss present a theoretical framework explaining contrastive learning's efficacy with simple augmentations and noise images by showing it learns partial whitening of the image power spectrum. Their analysis demonstrates that optimal representations are achieved by shallow convolutional networks utilizing sinusoidal filters.

Key insights

  • For stationary signals and simple augmentations, optimal contrastive learning performs partial whitening on the image's power spectrum, by measuring squared DFT coefficients scaled inversely to their expected power.
  • This optimal representation is realized by a shallow CNN with sinusoidal first-layer filters, a pointwise non-linearity, global average pooling, and a linear projection for partial whitening.
  • A "waterfilling" algorithm determines which frequencies are measured and their scaling, balancing data whitening with robustness to augmentation noise, concentrating sensitivity in mid-range frequencies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.