Reproduce “Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi-Lun Liao, Tess Smidt
Equiformer introduces an Equivariant Graph Attention Transformer architecture that integrates E(3)/SE(3) equivariance with Transformer operations for 3D atomistic graphs. The model achieves competitive or superior performance across QM9, MD17, and OC20 datasets, while demonstrating up to 15.5x faster training on large-scale tasks compared to previous models.
Key insights
- Transformer networks can generalize effectively to 3D atomistic graphs when seamlessly integrated with E(3) equivariance through equivariant operations and features based on irreducible representations.
- MLP attention and non-linear message passing offer enhanced expressivity and universal approximation capabilities for learning complex interactions between atoms, surpassing simpler dot-product attention in equivariant contexts.
- A straightforward architectural strategy of translating standard Transformer components into their equivariant forms, combined with efficient depth-wise tensor products, yields strong empirical results and improved computational efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.