Reproduce “BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sean Wu, Fredrik K. Gustafsson, Edward Phillips, Boyan Gao, Anshul Thakur, David A. Clifton

This research introduces the Behavioral Alignment Score (BAS), a decision-theoretic metric for evaluating the reliability of large language model confidence within an answer-or-abstain framework. It demonstrates that BAS offers a more robust assessment of decision-level reliability, particularly by severely penalizing overconfident errors, and shows that elicitation strategies like "top-k" and post-hoc calibration significantly enhance confidence trustworthiness across diverse LLMs and tasks.

Key insights

  • BAS serves as a proper decision-theoretic scoring rule, where truthful confidence uniquely maximizes expected utility, thus offering a principled measure for decision-making under uncertainty.
  • LLM confidence reliability is highly task-dependent; models exhibited positive BAS on structured tasks like MedQA but consistently negative BAS on open-ended factual QA like SimpleQA, indicating a tendency for overconfident incorrect answers when knowledge is missing.
  • Simple interventions such as "top-k elicitation" (prompting for multiple candidates) and post-hoc isotonic regression significantly improve BAS and reduce overconfidence, offering practical methods to enhance black-box LLM trustworthiness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.