Reproduce “Dropout in Training Neural Networks: Flatness of Solution and Noise Structure” on OpenResearch

About this paper

Zhongwang Zhang, Hanxu Zhou, Zhi-Qin John Xu

Investigating dropout's role in deep neural network training reveals it guides optimization towards flatter loss landscape minima, which correlates with improved generalization. This effect stems from dropout's structured, anisotropic noise, whose variance increases in sharper directions, steering the learning process away from less robust solutions.

Key insights

• Dropout regularization consistently promotes the discovery of flatter minima in the loss landscape, a characteristic empirically linked to better generalization performance across various neural network models and tasks.
• The noise introduced by dropout during training is not isotropic but highly anisotropic and structured, exhibiting a direct alignment with the local curvature (Hessian) of the loss landscape.
• This structured noise is instrumental in guiding the optimization process: it induces larger variance in sharper directions of the loss landscape, effectively perturbing the network parameters away from sharp, potentially poor-generalizing, minima.

Summary by alphaXiv.