Reproduce “Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala, Lenka Zdeborová
This research develops a high-dimensional asymptotic theory for single-head tied attention layers, providing an analytical framework that links generalization performance, learned weight matrix spectra, and neural scaling laws. The work predicts specific spectral patterns in attention weights, such as low-rank collapse and outliers, and shows how these properties relate to learning target features and contribute to power-law decay in excess risk with increasing data.
Key insights
- Weight decay on the `W` weights implicitly induces a nuclear norm regularization on the effective attention matrix `S = W^T W / sqrt(pd)`, favoring low-rank solutions and explaining the superior performance of factorized parameterization.
- The learned singular-value spectrum of the query-key map (`W^T W`) is analytically characterized, revealing that isolated spectral outliers correspond to genuinely learned features, while the bulk reflects noise from finite samples, establishing a direct link between spectrum and generalization.
- Learning for power-law targets proceeds via sequential recovery of signal modes, where an effective regularization threshold shifts to incorporate progressively weaker spectral directions, providing a mechanism for emergence phenomena and universal power-law scaling in excess risk.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.