Reproduce “Stationary Activations for Uncertainty Calibration in Deep Learning” on OpenResearch

About this paper

Lassi Meronen, Christabella Irwanto, Arno Solin

Researchers from Aalto University developed a family of Matérn activation functions for deep neural networks, derived from Gaussian Process (GP) Matérn kernels using control theory principles. These new activations improve uncertainty calibration, particularly for out-of-distribution (OOD) data, allowing networks to provide more reliable confidence estimates, as demonstrated on classification benchmarks and a radar emitter task.

Key insights

• A direct and principled method exists to design neural network activation functions based on desired Gaussian Process kernel properties, bridging theoretical concepts from control theory and functional analysis.
• By adopting Matérn activations, neural networks can inherently exhibit more robust uncertainty calibration, particularly for OOD inputs, without requiring complex post-hoc detection methods.
• The smoothness parameter 'nu' of Matérn activations offers a tunable control over the learned function's differentiability and local stationarity, with lower 'nu' values often yielding better uncertainty estimates than infinitely smooth RBF.

Summary by alphaXiv.