Reproduce “The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Max Springer, Chung Peng Lee, Blossom Metevier, Jane Castleman, Bohdan Turbal, Hayoung Jung, Zeyu Shen, Aleksandra Korolova

Princeton University researchers developed a geometric theory revealing that alignment collapse in fine-tuned LLMs stems from the curvature of parameter space, where benign updates can systematically steer models into sensitive regions. This work establishes a quartic scaling law showing alignment degradation accelerates with the fourth power of training time.

Key insights

  • Alignment fragility is an intrinsic geometric property of gradient descent on curved manifolds, rather than solely a consequence of harmful data or adversarial intent.
  • Second-order curvature coupling in the fine-tuning loss systematically induces non-trivial acceleration into alignment-sensitive subspaces, bypassing first-order defenses and explanations.
  • The prevailing intuition of 'high-dimensional orthogonality' is structurally unstable, as alignment resides in small, sharply curved subspaces, making it inherently fragile under gradient descent.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.