Reproduce “SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingyue Cui, Linghui Shen, Xingyi Yang
This research reveals that behaviors suppressed by Sparse Autoencoder (SAE) interventions in large language models can often be recovered post-intervention, even when targeted SAE features remain clamped. The study demonstrates that models exploit alternative computational routes, particularly the SAE reconstruction residual, achieving high recovery rates such as 98.9% for unlearned knowledge with zero feature drift.
Key insights
- SAE features act as causal handles but are not "complete intervention bottlenecks"; suppressed behavior can recover from a defended state even when targeted features remain clamped.
- The SAE reconstruction residual, typically viewed as inert error, plays a dominant role in carrying and rerouting behaviorally sufficient information, allowing models to circumvent feature clamps.
- Current SAE-based safety evaluations, which primarily verify initial suppression, are insufficient and need to incorporate tests for resilience against post-intervention recovery.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.