Reproduce “Entropic Out-of-Distribution Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Macêdo, Tsang Ing Ren, Cleber Zanchettin, Adriano L. I. Oliveira, Teresa Ludermir
The paper introduces IsoMax loss, a new training objective that serves as a direct replacement for SoftMax loss to enhance out-of-distribution (OOD) detection capabilities in deep neural networks. It achieves improved OOD detection performance by promoting isotropic feature learning and higher-entropy probability distributions without compromising in-distribution classification accuracy, increasing inference speed, or requiring additional data or hyperparameter tuning.
Key insights
- The inherent properties of SoftMax loss, specifically its anisotropy and tendency to produce overconfident predictions, fundamentally limit a deep neural network's ability to perform reliable out-of-distribution (OOD) detection.
- Designing a loss function that incentivizes isotropic, distance-based feature learning allows models to naturally cluster in-distribution data and separate OOD examples without external data or post-processing.
- Applying an 'entropy maximization trick' during training, by scaling distances, effectively trains models to produce high-entropy (more uncertain) posterior probability distributions at inference, which correlates directly with improved OOD detection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.