Reproduce “Latent Adversarial Training Improves the Representation of Refusal” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexandra Abbas, Nora Petrova, Helios Ael Lyons, Natalia Perez-Campanero
Researchers demonstrate that Latent Adversarial Training (LAT) concentrates the representation of refusal behavior in language models' latent space, with the first two SVD components explaining 75% of activation differences variance while producing more transferable refusal vectors compared to traditional safety fine-tuning approaches.
Key insights
- LAT concentrates refusal representation more prominently in first two SVD components (54% in first component alone)
- Layer 14 consistently shows highest effectiveness for refusal direction ablation across model variants
- More concentrated representations lead to both stronger and more vulnerable refusal mechanisms
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.