Reproduce “SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dara Bahri, Heinrich Jiang, Yi Tay, Donald Metzler
SCARF introduces a self-supervised contrastive learning method tailored for tabular data, utilizing random feature corruption from empirical marginal distributions to generate input views. It consistently enhances classification accuracy across fully-supervised, label noise, and semi-supervised settings, achieving average relative gains of 1-4% on 69 real-world datasets and demonstrating complementarity with existing supervised techniques.
Key insights
- Generating corrupted views by sampling from empirical marginal distributions of features is a highly effective and robust strategy for tabular data contrastive learning, surpassing other corruption methods.
- A dedicated pre-training phase using the contrastive InfoNCE loss is crucial for SCARF's effectiveness, outperforming strategies that integrate the loss during supervised training or use corruption solely for augmentation.
- SCARF exhibits stability across various hyperparameters like corruption rates and temperature, and is less sensitive to extremely large batch sizes, indicating its practical robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.