Reproduce “Hierarchies of Calibration: Classification meets Regression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Johannes Resin, Lu Yang, Tilmann Gneiting
Researchers from Goethe University Frankfurt, Heidelberg Institute for Theoretical Studies, and University of Minnesota systematically review, extend, and bridge various notions of calibration for probabilistic predictions in both classification and regression tasks. The work establishes rigorous hierarchical relationships among these diverse calibration concepts, providing a unified framework for understanding their interdependencies across different outcome types.
Key insights
- Auto-calibration represents the strongest form, implying most other calibration notions, but many weaker notions (e.g., confidence, marginal, modal, quantile calibration) are shown to be logically distinct and independent in general settings.
- As the outcome space simplifies from real-valued to binary, many previously distinct calibration notions converge and become equivalent, simplifying the assessment landscape.
- A generalized framework of "T-calibration," based on distributional functionals, unifies various specific calibration concepts, providing a clearer theoretical underpinning for their interdependencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.