Reproduce “From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pengyu Lai, Yixiao Chen, Dewu Yang, Rui Wang, Feng Wang, Hui Xu

DynFormer introduces a Transformer architecture for Partial Differential Equations that integrates principles of complex dynamics to handle multi-scale phenomena with enhanced efficiency and accuracy. The model achieves up to a 95.1% error reduction on chaotic systems like Kuramoto-Sivashinsky and reduces computational complexity from O(N^4) to O(N^3) for 2D problems.

Key insights

  • Embedding the 'slaving principle' of complex dynamics into the neural architecture allows small-scale dynamics to be implicitly determined by macroscopic states, avoiding explicit and computationally expensive global attention on fine scales.
  • Factorizing the attention mechanism into Kronecker products, which assumes separable dynamics for large-scale components, dramatically improves computational efficiency while maintaining accuracy for low-frequency interactions.
  • The multiplicative nature of the Local-Global-Mixing (LGM) transformation is crucial for expanding the spectral bandwidth of the output, enabling the reconstruction of high-frequency turbulent cascades that are essential for physical fidelity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.