Reproduce “Group Equivariant Stand-Alone Self-Attention For Vision” on OpenResearch

About this paper

David W. Romero, Jean-Baptiste Cordonnier

This formulation provides a general method to instill group equivariance into self-attention mechanisms by leveraging positional encodings that are invariant to specific symmetry group actions. The resulting Group Equivariant Stand-Alone Self-Attention Networks (GSA-Nets) demonstrate consistent performance improvements over non-equivariant baselines on vision benchmarks and offer inherent steerability for precise transformations.

Key insights

• Group equivariance in self-attention can be precisely imposed by defining positional encodings that are invariant to the action of the target symmetry group, thus enabling general and unique G-equivariance.
• GSA-Nets offer inherent steerability, as group actions directly manipulate continuous positional encodings, allowing for arbitrary transformations without discrete approximations or interpolation artifacts.
• Group self-attention generalizes group convolution and, when applied globally, can function as an equivariant universal approximator, expanding its potential expressiveness.

Summary by alphaXiv.