Reproduce “Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peter Súkeník, Marco Mondelli, Christoph Lampert
Researchers at ISTA developed the Deep Unconstrained Features Model (DUFM) and rigorously proved that Deep Neural Collapse (DNC) is the unique global optimum for multi-layer, non-linear networks under L2 loss, explaining this phenomenon for arbitrary depth without prior limiting assumptions. Their findings are supported by numerical experiments showing gradient descent convergence to these optimal DNC states.
Key insights
- Deep Neural Collapse (DNC) is the unique global optimum for the DUFM when trained with L2 loss and regularization, providing a foundational explanation for its emergence in deep networks.
- The specific DNC properties (within-class collapse, orthogonal class-means, aligned weights) manifest differently across layers, with the first two layers showing nuances due to nuclear norm minimization.
- Gradient descent successfully converges to the theoretically predicted DNC solutions in DUFM, and practical DNNs can produce features that align with DUFM's DNC predictions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.