Reproduce “Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup” on OpenResearch

About this paper

Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborová

The authors analyze how student-network width affects generalisation in teacher–student regression trained by online stochastic gradient descent (SGD). They track teacher–student weight overlaps with differential equations, proven to approximate SGD asymptotically under bounded smooth-activation assumptions. Wider sigmoid students with both layers trainable show lower asymptotic error, while fixed-output-weight models can worsen with width; the findings are specific to this setup.

Key insights

• The generalization properties of neural networks are not solely an intrinsic feature of SGD, but rather stem from a complex interplay between the optimization algorithm, the specific network architecture (including activation functions), and which layers are trained.
• Sigmoidal networks, when both layers are trained, can leverage over-parameterization to form 'denoising solutions' where multiple student units specialize to a single teacher, effectively averaging noisy estimates to reduce generalization error.
• In contrast to sigmoidal networks training both layers, over-parameterization either leads to a linear increase in generalization error (Soft Committee Machines) or results in a constant error (linear/ReLU networks with both layers trained), indicating varied impacts of redundancy based on architectural choices.

Summary by alphaXiv.