Reproduce “Self-Supervised Learning with Kernel Dependence Maximization” on OpenResearch

About this paper

Yazhe Li, Roman Pogodin, Danica J. Sutherland, Arthur Gretton

SSL-HSIC learns image representations by assigning each unlabeled image an identity shared by transformed views, maximizing kernel-based dependence on that identity while penalizing kernelized variation. Random Fourier features make the estimator linear in batch size for a fixed configuration. On ImageNet, ResNet-50 reaches 74.8% top-1 with a target network, versus BYOL’s 74.3%, while SwAV reaches 75.3%, so it does not outperform every baseline.

Key insights

• Kernel-based dependence measures like HSIC inherently incorporate geometric information in the feature space, overcoming limitations of mutual information for representation learning.
• The SSL-HSIC objective intrinsically prevents representational collapse through its centering property, offering a principled alternative to architectural safeguards in negative-free methods.
• Existing successful self-supervised learning losses, such as InfoNCE, can be theoretically interpreted as implicitly approximating kernel-based dependence maximization with a variance-based regularizer.

Summary by alphaXiv.