Reproduce “Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Olli Tuomi
The authors show that a corpus-mean direction in transformer residual streams helps set MLP gates’ resting operating point. They decompose each gate’s mean pre-activation into coupling to the mean normalized MLP input and explicit bias, then edit that direction. In Phi-2 at layer 10, removal raises firing above 0.5 by 0.086, and OPT is an exception.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.