Reproduce “Searching the Search Space of Vision Transformer” on OpenResearch

About this paper

Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, Haibin Ling

Researchers at Stony Brook University and Microsoft Research Asia introduced S3, a method that automates Vision Transformer architecture design by simultaneously evolving the search space and discovering optimal models. The resulting S3 models achieved higher performance on ImageNet classification, with S3-T reaching 82.1% Top-1 accuracy, and demonstrated strong transferability to various downstream vision tasks.

Key insights

• The third stage of Vision Transformers is critical for performance, with the search process consistently increasing the number of blocks in this stage.
• Optimal window sizes for ViTs progress from smaller in shallow layers to larger in deeper layers, facilitating a gradual increase in receptive field.
• MLP ratios were found to progressively increase with network depth, suggesting a non-uniform distribution across transformer stages.
• Deeper layers often benefit from Q-K-V dimensions slightly smaller than embedding dimensions, potentially due to redundancy among attention heads.

Summary by alphaXiv.