Reproduce “AutoFormer: Searching Transformers for Visual Recognition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Minghao Chen, Houwen Peng, Jianlong Fu, Haibin Ling

AutoFormer, developed by Stony Brook University and Microsoft Research Asia, presents an automated method for searching optimal vision transformer architectures. It introduces a "weight entanglement" strategy for efficient supernet training, enabling the discovery of high-performing subnets that outperform manually designed transformers on ImageNet classification and transfer learning tasks.

Key insights

  • Weight entanglement, where common parts of different transformer blocks share weights, allows for faster convergence and higher-quality supernet training compared to traditional weight sharing.
  • Automated neural architecture search can discover vision transformer architectures that achieve better performance and efficiency trade-offs than manually designed counterparts.
  • Optimal vision transformer architectures often feature moderate depths (14-15 layers), larger embedding dimensions, an MLP ratio around 5, and typically 6 attention heads.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.