Reproduce “Implicit variance regularization in non-contrastive SSL” on OpenResearch

About this paper

Manu Srinath Halvagal, Axel Laborieux, Friedemann Zenke

Researchers from the Friedrich Miescher Institute for Biomedical Research and University of Basel provide a theoretical analysis of non-contrastive self-supervised learning, explaining how asymmetric losses implicitly regulate representation variance under both Euclidean and cosine metrics. They introduce Isotropic Loss (IsoLoss), which equalizes learning dynamics across representational modes, leading to faster initial learning and stable training even without an exponential moving average target network.

Key insights

• Both Euclidean and cosine asymmetric losses implicitly regulate representation variance along orthogonal eigenmodes, preventing collapse by driving predictor eigenvalues towards stable non-zero values.
• Euclidean and cosine losses exhibit distinct learning dynamics: Euclidean loss forces predictor eigenvalues towards one, while cosine loss drives them to equal but arbitrary non-zero magnitudes.
• Predictor eigenvalues act as multiplicative factors for learning rates, creating an anisotropy where larger eigenvalues converge faster.

Summary by alphaXiv.