Reproduce “Rethinking Gauss-Newton for learning over-parameterized models” on OpenResearch

About this paper

Michael Arbel, Romain Menegaux, Pierre Wolinski

The authors study exact Gauss–Newton training in synthetic teacher-generated regression with one-hidden-layer student networks. They find that small hidden-weight initialization and small step sizes can improve learned features, even when the model’s losses remain high because its output layer lags. Freezing the features and refitting the linear readout reveals this improvement, which is limited to feature quality in these experiments.

Key insights

• Gauss-Newton dynamics can achieve global linear convergence for over-parameterized non-linear networks in the mean-field regime, provided specific initialization conditions (diverse hidden features, almost-optimal initial linear layer) are met.
• GN exhibits a distinct implicit bias compared to GD; smaller step sizes for GN lead to superior generalization by learning higher-quality features, implying a trade-off between optimization speed and generalization.
• A 'hidden learning phenomenon' demonstrates that high-quality internal features can be learned by GN even when the overall model's training and test losses are temporarily high due to an under-optimized final output layer.

Summary by alphaXiv.