Reproduce “Massive Activation Gating Channel in Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Minjia Mao, Shi Chen, Bowen Yin, Xiao Fang

The authors identify a massive activation gating channel (MAGC) that controls unusually large feed-forward network (FFN) activations in decoder-only language models. They show that changing this coordinate selects a concentrated mixture of down-projection directions, whose channel pattern the FFN output inherits. Across six models and four families, the mechanism is demonstrated at inference time; its training origin remains unresolved.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.