Reproduce “Choose a Transformer: Fourier or Galerkin” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuhao Cao
Researchers developed Fourier and Galerkin Transformer architectures for data-driven operator learning, which achieve improved accuracy and computational efficiency compared to prior models in solving forward and inverse PDE problems, including those with noisy measurements.
Key insights
- The softmax normalization in scaled dot-product attention is sufficient but not necessary for the Transformer's approximation capacity, as demonstrated by the theoretical soundness of linearized variants.
- A linearized Galerkin-type attention mechanism can explicitly represent a Petrov-Galerkin projection, with an approximation capacity proven to be independent of the sequence length.
- Carefully designed architectural elements like the new layer normalization and initialization schemes are essential for stabilizing and boosting the performance of softmax-free Transformers in scientific computing tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.