Reproduce “On the duality between contrastive and non-contrastive self-supervised learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, Yann Lecun
Researchers at Meta AI and collaborating institutions established an algebraic duality between sample-contrastive and dimension-contrastive self-supervised learning objectives, demonstrating that these seemingly distinct methods learn fundamentally similar representations and that previously observed performance differences can be mitigated through hyperparameter optimization and projector design.
Key insights
- An algebraic duality exists between sample-contrastive and dimension-contrastive loss criteria, implying they are equivalent objectives up to normalization terms related to embedding matrix row and column norms.
- The specific form of the repulsive force (LogSumExp vs. sum of squares) and explicit normalization strategies (centering, standardization) have limited practical impact on peak performance, suggesting flexibility in loss function design.
- Projector network capacity and meticulous hyperparameter tuning (e.g., temperature) are critical factors that significantly influence performance across both contrastive and non-contrastive methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.