Reproduce “Conformal Prediction with Macro-Coverage Guarantees” on OpenResearch

About this paper

Aabesh Bhattacharyya, Tiffany Ding, Rina Foygel Barber

Researchers from the University of Chicago and UC Berkeley introduced a label-weighted conformal prediction framework that provides finite-sample generalized macro-coverage guarantees, offering a practical balance between marginal and class-conditional coverage in long-tailed classification. The method, particularly when utilizing its proposed optimal score function, reliably achieved target macro-coverage while producing more informative prediction sets compared to existing approaches on large-scale image classification datasets.

Key insights

• Finite-sample generalized macro-coverage can be reliably achieved by adapting weighted conformal prediction with a correction factor that accounts for the number of calibration examples in each group.
• The most informative prediction sets (smallest size) for a given generalized macro-coverage objective are formed using a score function that incorporates class probabilities and group-specific weights.
• Allowing weights to adapt based on observed calibration data (H-measurable weights) enhances flexibility and mitigates issues arising from sparsely represented or unobserved groups.

Summary by alphaXiv.