Reproduce “Where Steering Signals Come From: Activation Source Selection in Activation Steering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaran Ye, Lingxu Ran, Zijun Yao, Chenpeng Wang, Yong Jiang, Lei Hou, Juanzi Li, Liangming Pan
The authors study activation source selection for additive steering, varying both source context and hidden-state readout. Rather than reading a completed target-bearing answer, they read an execution-boundary state immediately before the behavior appears. On four behavioral task families, Qwen2.5-7B-Instruct reached 64.0% with hybrid boundary plus last-token versus 19.3% with answer-only plus sequence-mean, under a searched grid within three instruction-tuned open-weight models.
Key insights
- Pre-generation or execution-boundary states are generally more useful for inducing a behavior than states collected after a target answer has already been produced.
- Answer-based sources can still work when their averaged activations retain information about continuing a target style or behavior, rather than merely representing the completed answer.
- The strongest steering signal is often mixed with generic continuation semantics; subtracting the shared tail exposes a cleaner direction and improves multiple vector-construction methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.