Reproduce “Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, Bryan Catanzaro
Adaptive Fourier Neural Operators (AFNO) provides an efficient token mixing mechanism for Vision Transformers, addressing the quadratic complexity of self-attention. It achieves comparable or superior accuracy while significantly reducing computational and memory costs across various vision tasks, particularly for high-resolution inputs.
Key insights
- Token mixing can be efficiently performed as global convolution in the Fourier domain, leveraging the Fast Fourier Transform for quasi-linear computational complexity.
- Adaptive weight sharing and block-diagonal channel mixing significantly reduce parameter count and increase expressiveness and generalization compared to static or full-matrix Fourier methods.
- Sparsity in the Fourier domain, enforced through soft-thresholding, acts as an effective regularization mechanism for images, improving performance and robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.