Reproduce “Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty” on OpenResearch

About this paper

Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, Dawn Song

Self-supervised learning, specifically using image rotation prediction as an auxiliary task, enhances model robustness against adversarial attacks, common corruptions, and label noise, while also improving out-of-distribution detection capabilities. These benefits are observed even in settings with ample labeled data, without necessarily increasing clean classification accuracy.

Key insights

• Self-supervised learning acts as a powerful general-purpose regularization technique that intrinsically enhances fundamental model properties like robustness and uncertainty estimation, even when ample labeled data is present.
• The benefits of self-supervision extend beyond mitigating data scarcity or boosting clean accuracy, providing consistent improvements across diverse aspects of model reliability.
• Robustness to adversarial attacks, common corruptions, and noisy data, alongside accurate uncertainty estimates, are crucial new metrics for evaluating future self-supervised learning algorithms.

Summary by alphaXiv.