Reproduce “Attention (as Discrete-Time Markov) Chains” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yotam Erel, Olaf Dünkel, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, Amit H. Bermano

Transformer attention matrices are reinterpreted as Discrete-Time Markov Chains (DTMC) to model indirect token interactions, introducing "TokenRank" for global token importance and "multi-bounce attention" for refined visualizations. This approach enhances zero-shot segmentation to 94.29 mAP and improves unconditional image generation, offering a deeper understanding of attention dynamics.

Key insights

  • Attention mechanisms in transformers can be effectively modeled as DTMCs, providing a unified theoretical framework that reinterprets existing operations and enables the derivation of new ones.
  • Indirect attention propagation through multiple "bounces" consolidates semantically similar regions into "metastable states," filtering noise and yielding sharper, more meaningful attention maps.
  • The steady-state vector (TokenRank) of the attention DTMC offers a globally informed and more faithful measure of token importance than direct attention scores or simple aggregations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.