Reproduce “Learning Optimal Conformal Classifiers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Stutz, Krishnamurthy, Dvijotham, Ali Taylan Cemgil, Arnaud Doucet
The authors train classifiers for conformal prediction sets by differentiating through simulated calibration and set prediction. After training, standard conformal prediction uses a held-out calibration set. On Fashion-MNIST, a two-layer MLP achieved average set size 1.67 versus 2.05 for cross-entropy training at 0.01 miscoverage, with marginal coverage requiring exchangeable calibration and test data.
Key insights
- Making the core steps of conformal prediction (set construction and calibration threshold determination) differentiable allows the entire conformalization process to be integrated into a neural network's training loop for end-to-end optimization.
- Optimizing model parameters directly with respect to desired confidence set properties, rather than just classification accuracy, can yield more efficient and application-specific uncertainty estimates.
- Maintaining theoretical marginal coverage guarantees is achievable by using smooth approximations only during training and reverting to the original, non-smooth CP methods for final inference on a held-out calibration set.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.