Reproduce “S2AFormer: Strip Self-Attention for Efficient Vision Transformer” on OpenResearch

About this paper

Guoan Xu, Wenfeng Huang, Wenjing Jia, Jiamao Li, Guangwei Gao, Guo-Jun Qi

The S2AFormer architecture features a Strip Self-Attention (SSA) mechanism that jointly compresses spatial and channel dimensions for efficiency, paired with a Local Interaction Module (LIM) for enhanced local feature representation. This design yields high accuracy on classification, segmentation, and detection tasks while demonstrating superior inference speeds across GPU, CPU, and ONNX platforms.

Key insights

• The Strip Self-Attention (SSA) mechanism can significantly reduce computational complexity (from O(N^2) to near linear) by simultaneously compressing spatial and channel dimensions, while effectively maintaining global contextual understanding.
• Integrating a Local Interaction Module (LIM) effectively addresses the limitations of purely attention-based mechanisms in capturing fine-grained local features and maintaining geometric invariance.
• A hybrid architectural approach, combining efficient global attention with dedicated local interaction, achieves a superior balance between model accuracy and computational efficiency.
• Convolution-based spatial reduction within attention mechanisms proves more effective for downsampling feature maps, leading to better accuracy, lower MACs, and faster inference compared to pooling methods.

Summary by alphaXiv.