Reproduce “Confidence Estimation in Automatic Short Answer Grading with LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne

A study introduces a hybrid confidence framework for automatic short answer grading using large language models, combining verbalizing, latent, and consistency-based model signals with dataset-derived semantic heterogeneity. This approach yields a selective prediction AUROC of 0.885 and significantly improves calibration, providing more reliable estimates for human-in-the-loop educational assessment.

Key insights

  • Model-based confidence signals alone (verbalizing, latent, consistency) showed limited selective prediction performance and exhibited miscalibration, particularly overconfidence in mid-confidence ranges.
  • Latent-based confidence from LLMs performed poorly, suggesting raw token-level probabilities primarily reflect decoding determinism rather than true classification uncertainty in generative models.
  • Integrating dataset-derived aleatoric uncertainty, represented by semantic heterogeneity, with model-based uncertainty through a hybrid fusion significantly improves both selective prediction and calibration of confidence scores.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.