Reproduce “Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah Smith
An investigation into transformer training dynamics reveals that the L2 norm of parameters grows proportionally to the square root of training steps, a phenomenon theoretically proven to lead to network saturation. This saturation biases attention heads towards specialized argmax-like and mean-like computational patterns, clarifying an inductive bias relevant to linguistic processing.
Key insights
- Parameter norm growth during gradient descent serves as a fundamental inductive bias, mechanistically steering transformer models towards a saturated computational regime.
- Saturated transformers develop distinct attention mechanisms, exhibiting bimodal patterns corresponding to argmax-like (local focus) and mean-like (global averaging/counting) operations.
- The observed square root (√t) norm growth is explained by the approximate 1-homogeneity of transformer encoders combined with misaligned gradient dynamics under common learning rate schedules.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.