Reproduce “Language Models Encode the Contextual Truth of Propositions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rupak Sarkar, Pritika Ramu, Rachel Rudinger
Researchers at the University of Maryland demonstrated that Large Language Models encode "contextual truth"—truth determined solely by in-context evidence—linearly within their activation spaces. This internal representation is robust across varying output tasks and causally influences model behavior, enabling a distinction between superficial and genuine shifts in belief during multi-agent dialogues leading to sycophancy.
Key insights
- LLMs possess a robust, linear internal representation for contextual truth that is independent of specific output task instructions and transfers across different contextual settings.
- The identified contextual truth direction is causally linked to the model's output decisions, demonstrating that intervening on this direction can steer model behavior.
- The study differentiates between "performative sycophancy" (output changes, internal truth representation doesn't) and "representational sycophancy" (internal truth representation shifts), with explicit linguistic affirmation correlating with representational shifts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.