Reproduce “On Connected Sublevel Sets in Deep Learning” on OpenResearch

About this paper

Quynh Nguyen

A theoretical analysis of deep over-parameterized neural networks with piecewise linear activation functions demonstrates that their loss landscapes exhibit connected and unbounded sublevel sets, and the absence of "bad local valleys". These geometric properties provide an explanation for the empirical success of local search algorithms in finding good solutions, under conditions where hidden layer widths are sufficiently large relative to the number of training samples.

Key insights

• Sufficient over-parameterization, specifically adequate hidden layer width relative to the number of training samples, dramatically simplifies the loss landscape's global structure.
• The absence of "bad local valleys" implies that any region where an optimizer might get stuck still offers paths to parameter configurations with arbitrarily low loss values, approaching the global infimum.
• While connected sublevel sets facilitate optimization, they do not inherently guarantee convergence to a global minimum, as saddle points and certain unbounded descent paths can still prevent it.

Summary by alphaXiv.