Reproduce “Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jihai Zhang, Xiang Lan, Xiaoye Qu, Yu Cheng, Mengling Feng, Bryan Hooi
The Multistage Contrastive Learning (MCL) framework addresses feature suppression in self-supervised learning by progressively learning overlooked features while retaining established knowledge. It demonstrated improved performance across unimodal and multimodal tasks, notably boosting average MMVP benchmark scores for CLIP from 20.0 to 32.6 with a ViT-L-14 encoder.
Key insights
- MCL progressively discovers and learns previously suppressed features (e.g., shape in Trifeature, CIFAR features in CIFAR-MNIST) while effectively preserving features already well-captured.
- The framework is model-agnostic, integrating with various contrastive learning methods (SimCLR, CLIP) and encoder architectures (ResNet, ViT) in both unimodal and multimodal settings without significant computational overhead.
- Each training stage of MCL learns distinct and non-redundant feature distributions, as evidenced by low Adjusted Mutual Information scores between clusterings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.