Reproduce “Interpreting Language Models with Contrastive Explanations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kayo Yin, Graham Neubig
Researchers from Carnegie Mellon University introduce contrastive explanations for language models, demonstrating they improve the identification of relevant linguistic evidence for complex decisions and enhance human understanding of model predictions. Their method, applied to GPT-2 and GPT-Neo, also enables clustering of foils to reveal how models make various fine-grained linguistic distinctions.
Key insights
- Contrastive explanations are more effective at identifying specific, linguistically relevant input features, particularly for capturing long-range dependencies, compared to non-contrastive methods.
- Providing contrastive explanations improves human observers' ability to predict and understand language model behavior, indicating that disentangling evidence by contrasting choices aids human simulatability.
- Clustering foils based on contrastive explanations reveals coherent linguistic distinctions made by LMs and the specific contextual features they leverage, offering insights beyond simple semantic similarity derived from word embeddings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.