Reproduce “Towards interpreting any residual stream direction by scaling activation inversion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger, Tomáš Gavenčiak, Gavin Leech, Euan Ong, Jiahai Feng
MAEM generates text from a residual-stream direction and checks whether the subject model’s activations on that text align with it. Its reward uses the subject model, not a human or learned judge. On held-out layer-42 Qwen3.6-27B targets, estimated best-of-eight centred cosine is 0.8219 versus 0.5720 for corpus search, showing stronger activation alignment in this setup, not semantic completeness or correctness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.