Reproduce “Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuling Jiao, Wensen Ma, Defeng Sun, Hansheng Wang, Yang Wang

DM trains image encoders to keep augmented views of one image close while matching feature distributions to a reference of concentrated components around orthogonal directions. On CIFAR-100, increasing reference components from 32 to 384 raises linear-probe accuracy from 45.78% to 67.71% and 5-nearest-neighbour accuracy from 22.61% to 56.18%, consistent with finer structure but not proof of semantic subclasses.

Key insights

  • Specifying a target distribution offers a direct way to shape representation geometry, rather than relying only on constraints intended to avert collapse.
  • Using more reference components than downstream classes can retain finer-grained structure; on CIFAR-100, increasing components from 32 to 384 improved linear-probe accuracy from 45.78% to 67.71%.
  • The classification guarantees are conditional, and the finer-component extension requires additional assumptions about latent subcategories and their aggregation into observed labels.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.