Reproduce “Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hoang Cuong Nguyen, Mark Dras, Usman Naseem

The paper compares three post-training methods—ordinary supervised fine-tuning, reasoning-augmented fine-tuning, and ORPO—across three language models to examine refusal-related computations. It estimates refusal directions, uses activation patching and steering, and evaluates harmful and benign prompts. In Gemma-2-9B, ORPO reaches 0.0% StrongREJECT ASR but 31.6% XSTest over-refusal, illustrating a tradeoff limited to the tested models and objectives.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.