Reproduce “Cell-Based Representation of Relational Binding in Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qin Dai, Benjamin Heinzerling, Kentaro Inui
Researchers at Tohoku University and RIKEN AIP identified a Cell-based Binding Representation (CBR) within Large Language Models that encodes discourse-level relational bindings with a grid-like geometry. Causal interventions demonstrated this low-dimensional subspace is essential for tracking complex entity-relation pairs across multiple sentences, remaining robust to various discourse perturbations.
Key insights
- LLMs implicitly encode discourse-level relational binding information in a low-dimensional linear subspace with a distinct grid-like geometry, separating entity and relation indices.
- The CBR subspace is causally relevant for relational binding, as demonstrated by targeted manipulations directly influencing the model's attribute predictions.
- The CBR structure is robust to various discourse perturbations, like reordering or ablating relations, and generalizes across diverse discourse patterns and semantic domains, suggesting a fundamental mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.