Reproduce “Average gradient outer product as a mechanism for deep neural collapse” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel Beaglehole, Peter Súkeník, Marco Mondelli, Mikhail Belkin
Researchers at UC San Diego and ISTA propose the Average Gradient Outer Product (AGOP) as a data-dependent mechanism driving Deep Neural Collapse (DNC). They demonstrate its role in feature representation collapse in Deep Recursive Feature Machines and show its link to within-class variability reduction in standard deep neural networks.
Key insights
- The Average Gradient Outer Product (AGOP) acts as a data-dependent mechanism that actively drives the collapse of within-class variability (NC1) in feature representations.
- Deep Neural Collapse (DNC) can be viewed as an implicit bias of minimizing the predictor norm in a parameterized kernel learning objective.
- The right singular structure of weight matrices in standard DNNs, empirically aligned with the AGOP, is primarily responsible for reducing within-class feature variability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.