Reproduce “LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval” on OpenResearch

About this paper

Elliot L. Epstein, John Winnicki, Thanawat Sornwanee, Rajat Dwaraknath

Researchers at Stanford University introduced FermiEval, a benchmark for evaluating Large Language Model confidence interval calibration in numerical estimation tasks. The study found LLMs exhibit systematic overconfidence, with nominal 99% intervals achieving only approximately 65% coverage, and demonstrated that conformal prediction effectively restores calibration, reducing the WinklerScore by up to 59.4%.

Key insights

• LLMs consistently demonstrate systematic overconfidence in numerical estimation, with their stated confidence levels not matching actual coverage.
• Conformal prediction provides a robust and effective way to calibrate LLM confidence intervals, offering theoretical coverage guarantees without assumptions on the model's output distribution.
• The 'perception-tunnel theory' proposes that LLMs act as if reasoning over a truncated slice of their inferred probability distribution, explaining the observed stagnation of coverage well below nominal levels.

Summary by alphaXiv.