Reproduce “Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, Jovana Mitrovic
ReLICv2 learns image representations by matching differently augmented views while requiring similarity judgments to remain consistent. It combines masked-background views with multiple large crops and a few small crops. On ImageNet ResNet-50 linear evaluation, it reaches 77.1% top-1 accuracy, versus 75.6% for prior self-supervised leaders and 76.5% for the cited supervised baseline; labels train the evaluation classifier.
Key insights
- Explicitly penalizing spurious features through unsupervised saliency masking and strategic multi-view learning significantly enhances the quality of self-supervised representations.
- The combination of multiple large views to capture stylistic invariance and a small number of small views to promote occlusion robustness is crucial for superior representation learning.
- RELIC's explicit invariance loss plays a critical role in effectively leveraging these additional inductive biases, leading to substantial performance gains compared to purely contrastive methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.