Reproduce “Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Charles Jin, Martin Rinard
Researchers at MIT CSAIL developed Latent Causal Probing, a formal framework using structural causal models and causal mediation analysis to rigorously assess whether language models represent latent variables from their training data. Their empirical study on a synthetic grid-world task showed that LMs do induce these unobserved concepts, providing positive mediated measurements for both deductive knowledge and inductive generalization to unseen data.
Key insights
- Language models can learn and represent the latent causal structures of their training data, inferring unobserved concepts.
- Higher capacity probes (MLPs) can extract more of the encoded latent information from LMs, but a positive causal effect can still be demonstrated.
- Raw probing accuracy can be misleading, as high accuracy does not automatically imply that the LM causally encodes the information; mediated measurements are necessary for robust conclusions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.