Reproduce “The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks” on OpenResearch

About this paper

Daniel Kunin, Atsushi Yamamura, Chao Ma, Surya Ganguli

Researchers at Stanford University extended the maximum-margin bias theory to a broader class of "quasi-homogeneous" neural networks, demonstrating that gradient flow on exponential losses implicitly minimizes an asymmetric norm of only the highest-rate parameters. This asymmetric bias provides a theoretical explanation for the Neural Collapse phenomenon and can lead to a degradation in model robustness.

Key insights

• Gradient flow in quasi-homogeneous networks imposes an asymmetric implicit bias, preferentially minimizing the norm of only a subset of parameters (the highest-rate ones).
• This asymmetric implicit bias can degrade the robustness of models compared to those with a standard L2-norm minimization, as shown through analytical and simulated examples.
• The quasi-homogeneous maximum-margin bias provides a fundamental, universal mechanism that explains the Neural Collapse phenomenon observed in deep neural networks with normalization layers.

Summary by alphaXiv.