Reproduce “Spectral Degeneration of Self-Attention under Isotropic Score Geometry” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mengda Li, Jianfeng Yao
The authors analyze how row-wise softmax can compress weaker score directions in a canonical model using pre-softmax score singular values as token-position weights. Under spike–bulk separation, bidirectional attention has stable rank near k+1, while causal attention’s grows on the order of log context length. GPT-2 tests show similar spectral distributions, not matrix equivalence; score singular values alone do not determine attention spectra.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.