Reproduce “Hidden Dynamics of Massive Activations in Transformer Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jorge Gallego-Feliciano, S. Aaron McClendon, Juan Morinelli, Stavros Zervoudakis, Antonios Saravanos
Researchers quantitatively characterized the emergence and evolution of massive activations within transformer models across training, proposing a mathematical framework that models their dynamics with high accuracy. The study also demonstrated that key aspects of these activations, such as their steady-state levels and peak timing, are predictable from architectural parameters, offering a path to designing models with controlled activation behavior.
Key insights
- Massive activations are not present at initialization but emerge and evolve systematically throughout training, following predictable, architecture-dependent patterns.
- The evolution patterns of MAs differ significantly by layer depth, with shallow/deep layers showing an "early peak" then decay, while middle layers exhibit a monotonic "log increase" in magnitude.
- Key architectural features, such as attention density (attention heads per hidden dimension) and layer depth, serve as dominant controls over the steady-state magnitude and peak timing of massive activations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.