Reproduce “Reasoning-Finetuning Repurposes Latent Representations in Base Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jake Ward, Chuqiao Lin, Constantin Venhoff, Neel Nanda
Reasoning finetuning appears to repurpose latent representations already present in a base model to enable emergent behaviors like backtracking. Researchers demonstrated that steering vectors derived from a base model's activations can reliably induce backtracking in its finetuned counterpart, confirming that the base model contains the necessary, unexpressed representation.
Key insights
- Reasoning finetuning appears to repurpose pre-existing latent representations within a base model rather than constructing complex reasoning circuits from scratch.
- Steering vectors derived from base model activations, at optimal negative token offsets, can causally induce specific emergent behaviors like backtracking in finetuned models.
- The identified backtracking-inducing direction encodes an abstract concept beyond simple token-level boosts and is likely one of several components contributing to the full backtracking mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.