Reproduce “Calibration and Risk-Budgeted Decisions with System One Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Soundouss Messoudi, Youness Mansar
The authors apply Learn Then Test (LTT) to select thresholds for forced binary decisions, using a separate labeled calibration sample to certify when a system may take its useful action and otherwise requiring a conservative fallback. Across six benchmarks, LTT had fewer test-risk exceedances than Naive and Empirical selection, but returned fewer policies; its guarantee depends on i.i.d. calibration and deployment data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.