Reproduce “Selective Classification for Deep Neural Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yonatan Geifman, Ran El-Yaniv

A method for deep neural networks to selectively classify is presented, providing a statistical guarantee that the true error rate will not exceed a user-defined threshold with high probability, while maximizing prediction coverage. Using a VGG-16 architecture, this approach allowed for a 2% error rate in top-5 ImageNet classification to be guaranteed with 99.9% probability and approximately 53.5% test coverage.

Key insights

  • Integrating a known statistical generalization bound with a binary search over confidence thresholds enables precise, probabilistic control of selective risk in deep learning models.
  • Softmax Response (SR) proved to be a more effective and empirically superior confidence-rate function than Monte-Carlo Dropout for selective classification, especially on large-scale datasets like ImageNet.
  • The proposed method provides a mechanism for DNNs to achieve 'self-awareness' by acknowledging uncertainty, significantly enhancing their reliability in critical applications.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.