Reproduce “Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhicai Wang, Yanbin Hao, Xingyu Gao, Hao Zhang, Shuo Wang, Tingting Mu, Xiangnan He
A new Vision MLP architecture, PosMLP, incorporates Relative Positional Encoding (RPE) to efficiently parameterize cross-token relations. This design achieves competitive performance on ImageNet-1K and COCO2017, drastically reducing the token-mixing layer's parameter complexity from O(N^2) to O(1).
Key insights
- Explicitly parameterizing cross-token relations with Relative Positional Encoding significantly enhances Vision MLP performance and efficiency, reducing token-mixing complexity to O(1) using GGQPE.
- A hierarchical architecture combining convolutional components with RPE-enhanced MLP blocks effectively introduces local inductive bias and improves performance by over 4 percentage points.
- The bias term within the Positional Spacial Gating Unit (PoSGU) implicitly learns absolute positional information, performing comparably to explicit absolute positional encoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.