Reproduce “Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centering” on OpenResearch

About this paper

Nicholas Pochinkov, Ben Pasero, Skylar Shibayama

This research systematically investigates neuron ablation methods, demonstrating that the common practice of zeroing out activations can be suboptimal. The study proposes 'peak ablation,' which uses the modal activation value for replacement, and shows that this method often outperforms traditional zero ablation, particularly in causal language models.

Key insights

• Neuron activation distributions in transformer models are often complex, non-Gaussian, and not centered at zero, challenging the assumption that zero represents a neutral state.
• The optimal neuron ablation method can vary across different model architectures (e.g., causal language models vs. masked language models or vision transformers).
• Replacing neuron activations with their modal (most common) value, rather than zero, can more effectively neutralize a neuron's contribution while preserving model functionality.

Summary by alphaXiv.