Reproduce “CMT: Convolutional Neural Networks Meet Vision Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, Chang Xu

Researchers from the University of Sydney and Huawei Noah's Ark Lab introduce CMT, a hybrid neural network architecture that effectively combines Convolutional Neural Networks and Vision Transformers. This approach establishes a new state of the art in accuracy-efficiency trade-offs for image classification, object detection, and segmentation, with CMT-S achieving 83.5% top-1 accuracy on ImageNet with 4.0B FLOPs.

Key insights

  • Systematically integrating convolutional operations and a hierarchical structure into Transformer designs effectively addresses the limitations of pure Vision Transformers, such as their lack of local inductive bias and inability to generate multi-scale features.
  • The combination of a Local Perception Unit (LPU) and an Inverted Residual Feed-forward Network (IRFFN) within the Transformer block enhances local feature learning and efficiency with minimal computational overhead.
  • Lightweight Multi-head Self-attention (LMHSA), using depth-wise convolutions for key and value projection, significantly reduces the quadratic complexity of attention, making the model more scalable for higher resolutions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.