Reproduce “Revisiting the Calibration of Modern Neural Networks” on OpenResearch

About this paper

Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, Mario Lucic

Modern neural networks, particularly non-convolutional architectures like MLP-Mixer and Vision Transformers, achieve high accuracy while also being well-calibrated both in-distribution and under distribution shifts, challenging the prior belief that accuracy inherently leads to poor calibration. The study from Google Research indicates that architectural design, rather than just model size or pretraining duration, plays a crucial role in intrinsic calibration properties.

Key insights

• Contrary to prior findings, several recent model families, including non-convolutional architectures like MLP-Mixer and Vision Transformers, achieved both high accuracy and good calibration without requiring post-hoc recalibration.
• Model architecture, rather than simply model size or the amount/duration of pretraining, emerged as a primary determinant of a model's intrinsic calibration properties and its robustness to distribution shifts.
• While larger models showed a slight increase in in-distribution calibration error, they consistently improved classification accuracy and exhibited better calibration robustness under distribution shift.

Summary by alphaXiv.