Reproduce “EMP-SSL: Towards Self-Supervised Learning in One Training Epoch” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shengbang Tong, Yubei Chen, Yi Ma, Yann Lecun

EMP-SSL introduces an approach to dramatically accelerate self-supervised learning by processing numerous overlapping image patches to efficiently learn co-occurrence statistics. This method achieves competitive performance, such as 85.1% accuracy on CIFAR-10, in a single training epoch, significantly reducing computational requirements compared to traditional SSL.

Key insights

  • Self-supervised learning methods fundamentally learn the co-occurrence statistics of image patches.
  • Processing a large number of overlapping patches from a single image allows for efficient capture of co-occurrence statistics in a single training pass.
  • A balanced loss function combining invariance objectives with total coding rate regularization prevents representation collapse while encouraging meaningful feature learning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.