Reproduce “Evaluating and Recalibrating Probabilistic Classifiers from One-Bit Threshold Queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Roman Plaud, Antoine Saillenfest, Matthieu Labeau, Thomas Bonald, Willem Waegeman
The authors introduce one-bit auditing and recalibration for binary classifiers whose black-box services hide confidence scores. Each labeled example receives one query asking whether its hidden score exceeds a threshold, enabling calibration estimation and temperature scaling. Across four open-weight language models on BoolQ and MMLU, the method recovered over 98% of score-access temperature-scaling improvement; large-scale one-bit responses were simulated.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.