Reproduce “Truth-value judgment in language models: 'truth directions' are context sensitive” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Stefan F. Schouten, Peter Bloem, Ilia Markov, Piek Vossen
Researchers from Vrije Universiteit Amsterdam investigated the context sensitivity and causal role of "truth directions" in large language models' latent spaces, demonstrating that while these directions are context-sensitive and causally mediate inference, they often lack logical coherence when exposed to irrelevant information. The study also introduced Contrast Consistent Reflection (CCR) as a more stable unsupervised truth-value probing method.
Key insights
- Truth-value probes in LLMs are highly context-sensitive, but often lack logical coherence, exhibiting significant changes in truth-value assignment even when presented with irrelevant or corrupted contextual information.
- Truth-value directions identified in LLM latent spaces function as causal mediators in the inference process, actively influencing how in-context information is integrated into subsequent representations.
- Instruction-tuning affects the type of coherence errors observed, with instruction-tuned models showing a greater tendency towards marginal belief violations in later layers compared to pretrained-only models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.