Reproduce “Jailbreak Transferability Emerges from Shared Representations” on OpenResearch

About this paper

Rico Angell, Jannik Brinkmann, He He

Jailbreak transferability in Large Language Models is driven by shared contextual representations across diverse architectures. Researchers at New York University causally demonstrated that aligning models through benign-only distillation significantly increases vulnerability to existing and new attacks, while also showing some emergent safety improvements against specific attack types.

Key insights

• Jailbreak transferability emerges as an inherent consequence of shared contextual representations across models, rather than being an incidental flaw or artifact of safety training.
• End-to-end adversarial robustness, where safeguards are embedded in model weights, is fundamentally vulnerable to attacks that exploit representation alignment.
• External, modular safeguards (e.g., constitutional classifiers) may offer more robust defense mechanisms by decoupling safety enforcement from the core generative model.

Summary by alphaXiv.