Reproduce “Simple Mechanistic Explanations for Out-Of-Context Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Atticus Wang, Joshua Engels, Oliver Clive-Griffin, Senthooran Rajamanoharan, Neel Nanda
This study explores Out-of-Context Reasoning (OOCR) in fine-tuned large language models (LLMs), revealing that this complex generalization arises from the implicit learning or explicit application of a consistent steering vector to model activations. This mechanism successfully induces OOCR, including conditional behaviors like backdoors, demonstrating how simple internal modifications can lead to advanced generalization capabilities.
Key insights
- Out-of-Context Reasoning often reduces to the implicit learning or explicit application of a constant steering vector to model activations, simplifying its mechanistic explanation.
- Single-layer LoRA can be surprisingly effective for inducing OOCR, sometimes outperforming or matching full all-layer LoRA, particularly for tasks like 'Functions' and 'Locations'.
- An unconditional steering vector, applied consistently, can implement complex conditional behaviors like backdoors by leveraging existing query-key (QK) attention patterns in the base model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.