Reproduce “Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation” on OpenResearch

About this paper

Changcheng Li, Jiancan Wu, Hengheng Zhang, Zhengsu Chen, Guo An, Junxiang Qiu, Xiang Wang, Qi Tian

Researchers at the University of Science and Technology of China and Huawei Inc. developed CoCA, an end-to-end learning framework that enables large language models to provide a calibrated confidence score before generating an answer. This "confidence-first" approach jointly optimizes uncertainty estimation and answer accuracy, significantly reducing inference costs while enhancing model trustworthiness across various tasks.

Key insights

• Effective confidence estimation requires explicit optimization within a unified framework, as merely improving answer accuracy does not inherently yield calibrated uncertainty. Overconfidence is a common trait of instruction-tuned models.
• Dynamic confidence targets, based on the model's empirical success rate, are crucial for robust calibration, preventing the confidence predictor from overfitting static labels and ensuring it tracks the evolving model policy.
• Segment-specific reward assignment is vital to prevent "reward hacking," allowing stable joint optimization where both confidence calibration and answer quality improve without one sacrificing the other.

Summary by alphaXiv.