Reproduce “Targeted Neuron Modulation via Contrastive Pair Search” on OpenResearch

About this paper

Sam Herring, Jake Naviasky, Karan Malhotra

Contrastive Neuron Attribution (CNA) steers behavior by identifying MLP neurons whose activations differ between prompts showing and not showing a target behavior, then intervening on those neurons. In refusal experiments, ablating 0.1% of activations reduced refusals by more than 50% in seven of eight instruction-tuned models. CNA’s repetition-based quality metric stayed above 0.97, but does not establish fluency or helpfulness.

Key insights

• Alignment fine-tuning primarily transforms pre-existing late-layer discrimination structures in base models into functional, sparse refusal gates in instruct models, rather than creating entirely new structures.
• CNA allows for precise, neuron-level interventions that effectively modulate refusal behavior while consistently preserving output coherence and general model capabilities, unlike coarser residual-stream methods.
• The specific neurons implementing these discrimination functions are largely replaced during fine-tuning, even as their general late-layer location remains stable.

Summary by alphaXiv.