Reproduce “DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Thanh Nguyen, Trung Pham, Chaoning Zhang, Tung Luu, Thang Vu, Chang D. Yoo

DimCL introduces a self-supervised learning approach that applies contrastive learning along the feature dimension rather than between instances, promoting feature diversity. Integrated as a regularizer, it consistently improves performance of existing SSL frameworks, such as boosting BYOL's accuracy on CIFAR-10 from 91.7% to 93.1% and on ImageNet-1K from 72.5% to 73.3%.

Key insights

  • Applying contrastive learning along the feature dimension effectively promotes feature diversity and orthogonality within learned representations.
  • The hardness-aware property of the adapted InfoNCE loss is crucial, directing optimization towards challenging feature element pairs for more effective learning.
  • Enhancing self-supervised representations can be achieved by focusing on feature-level interactions (dimensional contrast) rather than solely instance-level contrasts.
  • Optimal performance is achieved with specific temperature ("0.1") and balance weight ("0.1") hyperparameter values, indicating a balance between feature diversity and overall loss contribution.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.