Reproduce “Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xianliang Xu, Ting Du, Wang Kong, Bin Shan, Ye Li, Zhongyi Huang
Researchers from Tsinghua University and Nanjing University of Aeronautics and Astronautics provide the first comprehensive theoretical convergence analysis for Natural Gradient Descent (NGD) when training over-parameterized two-layer Physics-Informed Neural Networks (PINNs). The study demonstrates NGD's linear convergence with an O(1) learning rate and quadratic convergence for smooth activation functions, alongside refining Gradient Descent convergence bounds for PINNs, and empirically shows NGD's superior accuracy and robustness.
Key insights
- NGD demonstrates significantly faster convergence for PINNs than GD, capable of operating with an O(1) learning rate that is independent of the Gram matrix's least eigenvalue and sample size.
- For PINNs utilizing smooth activation functions, NGD can achieve a quadratic convergence rate with a learning rate of 1, offering substantial speed improvements.
- The expected Gram matrix is strictly positive definite for a broad range of smooth activation functions, which is essential for the robust application of NGD in PINNs.
- Refined theoretical analysis for GD in PINNs provides less stringent requirements for learning rate and network width compared to previous work, improving understanding of its limitations and capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.