Reproduce “Neighborhood Attention Transformer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ali Hassani, Steven Walton, Jiachen Li, Shen Li, Humphrey Shi

This work introduces the Neighborhood Attention Transformer (NAT), a new hierarchical vision architecture built on Neighborhood Attention (NA), an efficiently implemented pixel-wise sliding window attention mechanism. The research demonstrates that NAT achieves competitive or superior performance across image classification, object detection, and semantic segmentation tasks compared to leading Vision Transformers and CNNs, often with improved computational efficiency.

Key insights

  • Explicit sliding window attention, when efficiently implemented with custom CUDA kernels (NAT T EN), can be computationally superior to established window-based attention mechanisms like Swin's.
  • Integrating pixel-wise local attention (Neighborhood Attention) with overlapping convolutional tokenizers effectively combines the beneficial inductive biases of CNNs (translational equivariance, spatial locality) with the power of Transformers.
  • The receptive field of Neighborhood Attention naturally grows with depth, eliminating the need for complex inter-window communication mechanisms found in other hierarchical Transformers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.