Reproduce “Improving Transformation Invariance in Contrastive Representation Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Adam Foster, Rattana Pukdee, Tom Rainforth

A gradient-based regularizer and test-time feature averaging are introduced to make contrastive representations explicitly less sensitive to nuisance transformations. Across CIFAR-10, CIFAR-100, the synthetic Spirograph dataset, and MoCo v2, the methods reduce transformation information in representations, improve linear transfer performance, and increase robustness to shifted augmentation distributions.

Key insights

  • Standard contrastive alignment can leave transformation information recoverable; explicitly minimizing representation sensitivity produces stronger invariance than relying on InfoNCE alone.
  • Training-time regularization and test-time feature averaging are complementary: the former targets differentiable transformations, while the latter aggregates across a broader set of views.
  • Invariance is useful only when the chosen transformations preserve task-relevant information, and the proposed gradient method remains local and incurs additional optimization cost.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.