Reproduce “Reported Confidence in LLMs Tracks Commitment More Than Correctness” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dharshan Kumaran
Large language models' verbal confidence reports are better characterized as signals of their "commit-readiness" rather than direct estimates of an answer's objective correctness, a finding distinct from how log-probability-based confidence aligns with accuracy. This re-evaluation has implications for interpreting and training LLM confidence mechanisms.
Key insights
- Verbal confidence reports in LLMs primarily reflect an internal "commit-readiness" state, predicting the model's behavioral decision to commit to an answer more strongly than the answer's objective correctness.
- Calibrated log-probability confidence, in contrast, largely functions as a first-order truth signal, with its ability to predict abstention closely coupled to its ability to discriminate correctness.
- Mechanistic analyses revealed that internal neural states in LLMs encode a strong, approximately orthogonal representation for the commitment decision, suggesting verbal confidence is a self-report of this decision-facing state.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.