Reproduce “Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxi Xia, Pedro Henrique Luz de Araujo, Klim Zaporojets, Benjamin Roth
A study systematically investigates the influences of response agreement, loss functions, and prompt styles on Large Language Model calibration across 12 diverse models and four datasets. It proposes and evaluates Calib-n, an auxiliary model framework, demonstrating that Focal Loss, few-shot prompts, and external calibration methods improve confidence alignment more effectively than relying on an LLM's internal or verbalized probabilities.
Key insights
- Auxiliary models consistently outperform LLM's inherent confidence metrics (internal probabilities and verbalized confidences) in terms of calibration.
- Focal Loss is the most effective optimization objective for training auxiliary models to improve LLM calibration.
- Few-shot prompting yields the best calibration performance for auxiliary models, indicating that better-structured prompts can provide stronger signals for external calibration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.