Reproduce “Discovery of Hidden Miscalibration Regimes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Katarzyna Kobalczyk, Mihaela van der Schaar
The authors introduce a method for discovering input-dependent overconfidence and underconfidence without predefined slices. It learns input-space neighborhoods by calibration behavior and estimates local signed residuals from labeled examples. On HH-RLHF, Qwen3-8B’s global smECE is 0.120, versus 0.151 and 0.253 in discovered overconfident and underconfident subsets. The method is a diagnostic, not a guarantee of local calibration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.