Reproduce “Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jae Oh Woo, Mengdie Flora Wang, Rahul Ghosh, Baishali Chaudhury, Mun Young Kim
The authors introduce a black-box audit of whether models can choose between two answers and report informative confidence. Three LLM judges provide a comparative reference, and the audit measures agreement and calibration by problem category. Across six models, JiuZhang3.0-7B leads on ECE and Brier but scores 0.019 on PM-C-PVC-VUS, versus 0.209 for Qwen2.5-32B-Instruct; this is a diagnostic, not a deployment safety test.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.