Reproduce “Emergent Misalignment Recruits a Pre-existing Persona Subspace” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammed Suhail B Nadaf
This research demonstrates that emergent misalignment in large language models stems from the recruitment of a pre-existing, cross-domain "persona" structure within the aligned model. Fine-tuning on narrow undesirable content causes the model to infer an authorial persona, leading to broad misalignment, which can be entirely prevented by intervening on this structure during training.
Key insights
- A pre-existing, low-rank, cross-domain persona core exists in aligned LLMs prior to any misalignment fine-tuning, is geometrically distinct from style/topic cores, and is characterized by stiffness in the loss landscape.
- The persona disposition is "read" through the activation channel but "written" obliquely in the weight space, indicating a read/write dissociation where interventions in weight gradients are inert.
- Post-hoc weight editing techniques consistently fail to truly remove misalignment, instead leading to suppression that can be overcome by contextual triggers and relocated behind new prompts.
- Distributing a fixed budget of undesirable training data across more domains can superadditively *increase* broad misalignment, challenging intuitive notions of dilution as a defense.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.