Reproduce “Emergence of meta-stable clustering in mean-field transformer models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Giuseppe Bruno, Federico Pasqualotto, Andrea Agazzi

This paper presents a detailed mathematical description of the internal dynamics of a simplified mean-field Transformer model, demonstrating how meta-stable phases of token clustering emerge and persist. It shows that these structured representations, which are crucial for the model's performance, last for a duration logarithmically dependent on the number of tokens (O(ln N)) and quantitatively links the number of emergent clusters to hyperparameters like the inverse temperature and embedding dimension.

Key insights

  • The emergence of structured, meta-stable multi-cluster representations in Transformer models is a predictable outcome of their internal dynamics, influenced by specific hyperparameters.
  • These meta-stable phases, which are crucial for representation learning, persist for a duration logarithmically dependent on the number of tokens (O(ln N)), making them highly relevant for large context windows.
  • The "richness" of representations (number of emergent clusters, denoted as k_max) is directly determined by the inverse temperature parameter (β) and embedding dimension (d).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.