Reproduce “Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems” on OpenResearch

About this paper

Tianle Cai, Ruiqi Gao, Jikai Hou, Siyu Chen, Dong Wang, Di He, Zhihua Zhang, Liwei Wang

Researchers at Peking University developed the Gram-Gauss-Newton (GGN) method, which reinterprets Gauss-Newton through the Neural Tangent Kernel (NTK) theory to create a computationally tractable second-order optimizer for overparameterized neural networks. This method demonstrated faster training convergence and superior test performance on regression tasks compared to Stochastic Gradient Descent (SGD).

Key insights

• The linear approximation inherent in the Gauss-Newton method aligns with the core principles of the Neural Tangent Kernel (NTK) in overparameterized networks, allowing for a connection between second-order optimization and kernel regression.
• Computational bottlenecks of second-order methods can be overcome by switching the matrix inversion from the large parameter-space Hessian approximation to a smaller data-space Gram matrix.
• Mini-batching can be effectively incorporated into a second-order method, providing the first theoretical convergence guarantee for such an approach in overparameterized neural networks.

Summary by alphaXiv.