Reproduce “Pure Transformers are Powerful Graph Learners” on OpenResearch

About this paper

Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong

This paper presents TokenGT, a method that enables standard Transformer architectures to learn from graph data by treating nodes and edges as tokens augmented with specific identifier embeddings. TokenGT outperforms message-passing GNNs and achieves competitive results against specialized graph Transformer variants on molecular property prediction tasks while being theoretically more expressive than existing message-passing GNNs.

Key insights

• Pure Transformer attention mechanisms can implicitly learn and interpret graph structure from appropriately tokenized inputs and auxiliary embeddings, without requiring explicit graph-specific architectural modifications.
• The use of orthonormal node identifiers in conjunction with type identifiers allows self-attention heads to approximate equivariant linear operators, providing the theoretical basis for TokenGT's high expressive power (equivalent to 2-Invariant Graph Networks).
• By encoding graph inductive biases into input embeddings rather than architectural components, TokenGT maintains compatibility with general Transformer optimizations (like linear attention), improving scalability.

Summary by alphaXiv.