Reproduce “Suppressing Misaligned Behaviors in Language Models, Durably” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han, Danqi Chen
The authors introduce PEARL, a post-hoc method that trains a model to recover from prefixes of its cached misaligned responses as well as respond safely from clean prompts. In broad-trigger insecure-code tests on GPT-OSS-20B, its SFT-warmup version had a 1.67% misalignment rate after 18,000 reactivation examples, below GRPO and SFT; on a narrow-trigger test, SFT was more durable.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.