Reproduce “MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of Physics-informed Neural Networks” on OpenResearch

About this paper

Jiachen Yao, Chang Su, Zhongkai Hao, Songming Liu, Hang Su, Jun Zhu

The MultiAdam optimizer introduces a parameter-wise scale-invariant approach for training Physics-informed Neural Networks, mitigating loss imbalance caused by domain scaling. This method consistently improves predictive accuracy by 1-2 orders of magnitude across diverse second-order PDEs, including Poisson, Helmholtz, and Burgers equations.

Key insights

• Domain scaling critically impacts PINN training by creating substantial loss imbalance, where PDE residual loss magnitude scales significantly with domain changes while boundary loss remains stable.
• Minimizing the L-infinity error of the solution requires optimizing a weighted sum of the square roots of PDE and boundary losses, with optimal weights directly influenced by domain scale.
• Parameter-wise re-scaling of gradients, informed by historical momentum estimates from individual loss components, effectively addresses loss imbalance and gradient pathologies in PINNs.

Summary by alphaXiv.