Reproduce “Large Language Model Failures From Hallucination to Homogenization Are Different Facets of Miscalibration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tiancheng Hu, Caiqi Zhang, Dirk Hovy, Nigel Collier
The authors present a diagnostic framework that treats biased sampling, factual overconfidence, narrowed outputs, and poor decisions about tools or abstention as overlapping forms of miscalibration. It distinguishes probabilistic, semantic, distributional, and metacognitive targets. On SimpleQA, frontier models reach around 50% accuracy, while stated confidence exceeded actual accuracy for every tested model; the framework does not establish a shared cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.