Reproduce “Rethinking Token-Mixing MLP for MLP-based Vision Backbone” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, Ping Li
Researchers at Baidu Research's Cognitive Computing Lab propose a Circulant Channel-Specific (CCS) token-mixing MLP for pure MLP-based vision backbones to address limitations in spatial invariance and channel expressiveness. This enhancement led to a 2.6 percentage point increase in ImageNet-1K top-1 accuracy for Mixer-B/16, reaching 79.8%, while simultaneously reducing model parameters.
Key insights
- Spatial invariance can be elegantly incorporated into MLP designs through mathematical constraints (circulant matrices) rather than complex architectural additions.
- Channel grouping enhances model expressiveness by allowing distinct spatial mixing patterns for different channel groups, efficiently capturing richer visual representations.
- The Fast Fourier Transform (FFT) based implementation reveals connections between spatial mixing in MLPs and frequency-domain processing, suggesting new avenues for understanding visual information processing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.