Reproduce “In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, Junxian He
This research investigates LLM hallucinations from an inner representation perspective, identifying "in-context sharpness" as a signal for factual correctness. It introduces Activation Decoding, an entropy-based constrained decoding method that consistently improves factuality and informativeness, achieving up to 8.6 absolute points increase in Truth*Info on TruthfulQA for LLaMA2-70B-chat.
Key insights
- Correct generations by LLMs are linked to significantly sharper and more focused activation patterns within the hidden states of in-context tokens.
- The proposed contextual entropy metric effectively quantifies this sharpness, achieving an AUROC of up to 0.76 in distinguishing between correct and incorrect answers on the GF-CFT dataset.
- Calibrating next-token predictions based on contextual entropy can reliably identify and favor more factual responses, providing a robust signal for hallucination mitigation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.