Reproduce “How Transformers Get Rich: Approximation and Dynamics Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingze Wang, Ruoxi Yu, Weinan E, Lei Wu
Researchers from Peking University rigorously analyzed how Transformer models approximate and learn "induction heads," a key mechanism for in-context learning (ICL). Their work provides architectural requirements for different induction head types and characterizes the four-phase training dynamics leading to their abrupt emergence.
Key insights
- Different Transformer submodules play distinct, specific roles in implementing induction heads: single heads for vanilla IH, multi-heads for generalized n-gram interactions, and FFNs for approximating arbitrary non-linear similarity functions in ICL.
- The abrupt emergence of induction heads is driven by a "time-scale separation" in parameter dynamics—dot-product components (for IH) have quadratic parameter dependence and slow initial ignition, unlike RPE components (for n-grams).
- Initial imbalances in the loss landscape, where induction head components have a smaller initial loss contribution, contribute to longer plateau phases before their rapid emergence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.