Reproduce “Confidence Is Not a Promise: A Reference Audit of Typed Decision Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pedro Carvalho Brom
The authors evaluate typed decision models that return probabilities over caller-supplied text options and test confidence-based gates. They compare threshold rules using calibration data, including Learn then Test and an ascending Bayesian Beta rule. For Portuguese MASSIVE intent questions, Jev 1.13 Learn then Test violated a 99% target in 1.4% of draws, versus 38.6% for the point rule, under the study’s empirical conditions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.