Reproduce “Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Model” on OpenResearch

About this paper

Onat Ure, Samet Demir, Zafer Dogan

This paper presents a generative data model offering precise control over high-order cumulants through Hermite polynomial expansion, allowing for systematic investigation into how non-Gaussian statistics influence neural network learning. The work demonstrates that neural networks learn data characteristics in a moment-wise progression, initially capturing low-order statistics before progressively exploiting higher-order cumulants to improve generalization.

Key insights

• Neural networks exhibit a "moment-wise progression" or "distributional simplicity bias," meaning they learn low-order data statistics before progressively incorporating higher-order cumulants.
• High-order cumulants, beyond mean and covariance, contain crucial statistical information that neural networks can leverage to enhance generalization performance.
• The Hermite polynomial expansion offers a principled and analytically tractable method to directly control and isolate the effects of specific cumulants on learning dynamics.

Summary by alphaXiv.