Reproduce “Toward a Graph Foundation Model: Pre-Training Transformers With Random Walks” on OpenResearch

About this paper

Ziyuan Tang, Jie Chen

Researchers at the University of Minnesota and MIT-IBM Watson AI Lab developed the Random Walk-Based Pre-Trained Transformer (RWPT), a foundation model that adapts the Transformer architecture for graph data by using random walks to represent nodes and a self-supervised context prediction objective for pre-training. RWPT demonstrates superior generalizability across diverse graph datasets and tasks, consistently outperforming existing graph deep learning methods and other foundation-model style approaches.

Key insights

• Random walks provide an effective sequential representation of graph structure for Transformers, enabling the capture of long-range dependencies and handling varying graph sizes.
• Combining a large language model for unified node features and shortest-path distances for positional encoding allows Transformers to effectively learn from diverse graph domains.
• A per-walk attention mask significantly improves the scalability of Transformer attention for graph data, while a self-supervised context prediction objective is effective for learning generalizable graph representations.

Summary by alphaXiv.