Reproduce “Pattern Formation in Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Erkan Turan, Gaspard Abel, Maks Ovsjanikov
The authors analyze how a Transformer’s architecture and initialization amplify token-pattern modes near the homogeneous state, using a frequency-by-frequency linear analysis of positional encoding, output–value geometry, and feed-forward dynamics. Controlled sequence experiments show task-aligned modes improve learning beyond matched gain alone. On CIFAR-10, weight-tied ConViT accuracy is 91.5 ± 0.2% versus 89.6 ± 0.4%, with evidence limited to these settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.