Reproduce “Split Conformal Classification with Unsupervised Calibration” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Santiago Mazuelas

Santiago Mazuelas, affiliated with BCAM and IKERBASQUE, developed a method for split conformal prediction that utilizes entirely unlabeled calibration samples. The approach assigns 'label weights' to these unsupervised samples based on supervised training data, achieving coverage probabilities comparable to conventional supervised methods, often within 0.01 of the target, while maintaining informative prediction set sizes.

Key insights

  • Leveraging the supervised training data to compute 'label weights' for unsupervised calibration samples enables reliable conformal prediction without requiring new labels.
  • The use of Integral Probability Metrics provides a theoretically grounded framework for making unsupervised calibration samples statistically indistinguishable from supervised ones, ensuring coverage guarantees.
  • Theoretical analysis demonstrates that the coverage gap decreases at a rate of O(1/sqrt(n) + 1/sqrt(m)), where 'n' is the number of calibration samples and 'm' is the number of training samples, with constants related to the smoothness of the conformal score.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.