Reproduce “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations” on OpenResearch

About this paper

Dan Hendrycks, Thomas Dietterich

The authors introduce ImageNet-C and ImageNet-P to measure classifier robustness in two ways: errors on corrupted images and prediction changes across sequences of small input perturbations. ImageNet-C tests 15 corruption types at five severities, while ImageNet-P tests ten perturbations. On ImageNet-P scale sequences, ResNet-18 flips its top-1 prediction between adjacent frames with 15.6% probability; these benchmarks do not estimate real-world failure rates.

Key insights

• Modern neural network architectures, despite achieving significantly higher accuracy on clean data, show limited or no intrinsic improvement in relative robustness to common corruptions when compared to older models like AlexNet.
• Current deep learning classifiers exhibit unexpectedly high prediction instability, with frequent changes in top-1 and top-5 predictions even under small, non-adversarial perturbations.
• Architectures that enhance feature aggregation and possess larger capacity (e.g., DenseNets, ResNeXts) consistently demonstrate improved robustness to both corruptions and perturbations, suggesting these benchmarks can indicate superior representation learning.
• Certain techniques, such as Adversarial Logit Pairing (ALP), initially designed for adversarial robustness, can unexpectedly provide substantial gains in robustness to common perturbations.

Summary by alphaXiv.