Reproduce “Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

George Ruellan, Diana Benavides-Prado

The authors compare supervised replay exposure with logit distillation during sequential fine-tuning. Exposure trains on supplied safe refusals and benign-compliance responses; distillation matches the base model’s output scores. On Mistral-7B, exposure yields 1.0% HarmBench attack success versus 46.7% for distillation, but OR-Bench-hard over-refusal is 66.4% versus 18.1%. These means cover three task orderings at seed 42, not general robustness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.