Reproduce “Asymptotic Behavior of Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Katy Blumer, Da-Cheng Juan, Jon Kleinberg, Ravi Kumar, Cyrus Rashtchian, Andrew Tomkins
The authors study indefinite generation by fixed autoregressive transformers, modeling accumulated history as a changing state. In a three-token attention-only transformer at temperature 1, with c greater than 6, empirical token frequencies fail to converge almost surely. For each fixed transformer, sufficiently high, model-dependent temperature yields almost-sure convergence of the key/value-cache distribution; no finite temperature guarantees convergence across the class.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.