Reproduce “Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tokio Kajitsuka, Issei Sato

Research from The University of Tokyo demonstrates that Transformers with just one self-attention layer, utilizing softmax and low-rank weight matrices, are universal approximators capable of contextual mapping and memorization. This work provides theoretical justification for the efficacy of shallow Transformer architectures, bridging the gap between existing deep-layer theoretical requirements and practical observations.

Key insights

  • The softmax function is a critical component for Transformer expressive power, enabling contextual mapping through its connection to the Boltzmann operator, unlike hardmax.
  • A single self-attention layer with low-rank (rank-1) weight matrices is sufficient for generating unique context IDs and achieving contextual mapping.
  • The theoretical expressive capacity of Transformers, including memorization and universal approximation, can be achieved with significantly shallower architectures than previously established.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.