Reproduce “An initial alignment between neural network and target is needed for gradient descent to learn” on OpenResearch

About this paper

Emmanuel Abbe, Elisabetta Cornacchia, Jan Hązła, Christopher Marquis

Researchers from EPFL introduce and prove the necessity of "Initial Alignment" (INAL), a measure of correlation between a neural network's initial state and a target function, for gradient descent to learn in polynomial time. Their work establishes that if a target Boolean function has negligible INAL with an expressive activation function, it cannot be weakly learned by polynomially-sized networks via noisy gradient descent.

Key insights

• Initial Alignment (INAL) serves as a quantifiable measure of how well a network's architecture and initialization are suited for a specific target function.
• A lack of sufficient INAL means the target function primarily consists of high-degree Fourier components, which are hard for standard neural networks to capture at initialization.
• The inability to find significant initial correlation (low INAL) fundamentally leads to unlearnability by gradient descent within practical computational limits, challenging the assumption that large networks can always learn.

Summary by alphaXiv.