Reproduce “Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks” on OpenResearch

About this paper

Guoxin Lu, Letian Sha, Qing Wang, Peijie Sun, Hao Zhou, Hua Dai, Fu Xiao

Researchers developed Safety Anchor, a defense mechanism for large language models that prevents the restoration of harmful capabilities during fine-tuning by anchoring refusal mechanisms at the unembedding layer. The approach achieved an average harmful score of 5.68% across various tasks, significantly outperforming existing defenses, while maintaining functional accuracy on benign tasks.

Key insights

• The unembedding layer acts as a deterministic 'geometric bottleneck' because controlling its input (the final hidden state) directly governs token probabilities, making it a robust control point for safety.
• Parameter redundancy allows existing defenses to be circumvented, as attackers can find 'orthogonal' paths in the high-dimensional parameter space that satisfy constraints but still restore harmful behaviors.
• Even a single safety anchor can be highly effective because diverse harmful intents converge to a shared latent representation, allowing localized stability to inhibit unseen attacks in that semantic neighborhood.

Summary by alphaXiv.