Reproduce “Minnorm training: an algorithm for training over-parameterized deep neural networks” on OpenResearch

About this paper

Yamini Bansal, Madhu Advani, David D Cox, Andrew M Saxe

Minnorm selects among fitting networks by minimizing per-layer weight norms subject to exact training fit or classification margin constraints. It updates weights by gradient descent and uses per-example error multipliers to adjust those updates; nonzero multipliers identify examples that contribute to the solution. On MNIST, it achieved 1.46 ± 0.063% test error versus 1.82 ± 0.056% for vanilla SGD, in the tested two-hidden-layer network and protocol.

Key insights

• Fixed-point analysis reveals that optimal minimum-norm weights are linear combinations of gradients weighted by Lagrange multipliers, identifying "support vector"-like critical examples.
• Minnorm training achieves exponentially faster convergence to max-margin solutions compared to the logarithmic convergence observed with traditional cross-entropy loss.
• It effectively prunes weights in "frozen subspaces" of under-determined deep linear networks, making generalization robust to initial weight scale and improving test error.
• Fixed-point analysis reveals that optimal minimum-norm weights are linear combinations of gradients weighted by Lagrange multipliers, identifying "support vector"-like critical examples.

Summary by alphaXiv.