Reproduce “SP-ViT: Learning 2D Spatial Priors for Vision Transformers” on OpenResearch

About this paper

Yuxuan Zhou, Wangmeng Xiang, Chao Li, Biao Wang, Xihan Wei, Lei Zhang, Margret Keuper, Xiansheng Hua

SP-ViT introduces learnable 2D spatial priors into Vision Transformers, enabling them to automatically discover diverse spatial relationships to enhance performance. The approach achieves 86.3% Top-1 accuracy on ImageNet-1K with nearly 50% fewer parameters than previous state-of-the-art models, alongside improved object localization in attention maps.

Key insights

• Allowing ViTs to *learn* spatial inductive biases from data, rather than relying on hard-coded designs, yields a more flexible and effective approach to integrate spatial reasoning.
• The multiplicative interaction between learned spatial priors and content scores effectively scales attention, prioritizing patches based on their spatial relationship while still considering semantic similarity.
• Assigning a unique learnable spatial prior to each attention head encourages the discovery of diverse, complementary spatial patterns (e.g., horizontal, vertical, non-local), leading to a richer overall understanding of image structure.

Summary by alphaXiv.