Reproduce “MetaFormer Baselines for Vision” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, Xinchao Wang

Researchers at NUS and Sea AI Lab investigate the MetaFormer architecture, demonstrating its inherent strength in vision tasks even with basic token mixers. Their CAFormer, combining convolutions and vanilla attention within this framework, achieves 85.5% top-1 accuracy on ImageNet-1K, establishing a new record using standard supervised training.

Key insights

  • The MetaFormer architectural template alone provides a robust baseline, achieving over 80% ImageNet-1K accuracy even with identity mapping as its token mixer.
  • MetaFormer is universally adaptable, performing well with arbitrary token mixing mechanisms, including random mixing, indicating its flexibility.
  • State-of-the-art vision model performance can be achieved effortlessly by integrating "old-fashioned" or conventional token mixers within the MetaFormer framework.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.