Reproduce “Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuntai Bao, Xuhong Zhang, Tianyu Du, Xinkui Zhao, Zhengwen Feng, Hao Peng, Jianwei Yin
Researchers from Zhejiang University empirically investigated "truth directions" in large language models, revealing that the consistency and generalizability of these internal representations of truth are strongly correlated with a model's overall capability. Their work demonstrates that simple linear probes can effectively identify truthfulness across various logical transformations, question-answering formats, and different knowledge sources, enhancing selective question-answering accuracy from 55.29% to 64.06% on TriviaQA for Llama-3.1-8B.
Key insights
- The presence and consistency of robust "truth directions" are emergent properties strongly correlated with an LLM's general capabilities, not a universal feature across all models.
- Simple linear probes are sufficiently expressive to identify truth directions in more capable LLMs where these features are distinctly represented, challenging the necessity of complex probing methods.
- Truthfulness representations generalize broadly, enabling probes trained on simple factual statements to effectively classify truth across logical transformations, diverse syntactic forms (statements to QA), and both parametric and in-context knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.