Reproduce “CAT: Cross Attention in Vision Transformer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hezheng Lin, Xing Cheng, Xiangyu Wu, Fan Yang, Dong Shen, Zhongyuan Wang, Qing Song, Wei Yuan
Researchers from KuaiShou Inc. and Beijing University of Posts and Telecommunications developed the Cross Attention Transformer (CAT), an architecture that effectively combines local and global attention within a hierarchical framework for computer vision. CAT achieves strong performance on image classification, object detection, and semantic segmentation benchmarks while maintaining high computational efficiency.
Key insights
- The alternating application of local (IPSA) and global (CPSA) attention within a hierarchical network structure effectively balances feature richness and computational efficiency.
- Performing self-attention on single-channel feature maps (CPSA) provides a computationally efficient method for capturing global relationships between distant image patches.
- Absolute position encoding is beneficial for providing global spatial context in CPSA, while the architecture's inherent design alleviates the need for complex 'shifted window' mechanisms often found in other hierarchical Transformers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.