Reproduce “Geometry-Calibrated Conformal Abstention for Language Models” on OpenResearch

About this paper

Rui Xu, Yi Chen, Sihong Xie, Hui Xiong

Researchers at the Hong Kong University of Science and Technology (Guangzhou) developed Conformal Abstention, a post-hoc framework enabling large language models to selectively abstain from answering queries, thereby mitigating hallucinations. This method provides finite-sample guarantees on both participation and conditional correctness, achieving an average conditional correctness of 75.0% for retained responses by leveraging internal token representation geometry for uncertainty calibration.

Key insights

• Correct LLM predictions are strongly associated with specific internal geometric dynamics, including robust knowledge contribution from MLP blocks coupled with significant embedding rotations, and closer alignment to in-distribution anisotropic cones in the embedding space.
• Leveraging internal token representation geometry provides a more faithful measure of LLM uncertainty and knowledge involvement compared to surface-level or simpler internal signals.
• Conformal Prediction can be effectively applied to open-ended generative tasks by defining abstention based on calibrated prediction confidence, circumventing the difficulty of traditional non-conformity scores.

Summary by alphaXiv.