Reproduce “Neural (Tangent Kernel) Collapse” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, Hung-Hsu Chou

This research theoretically connects Neural Tangent Kernel (NTK) alignment with Neural Collapse (NC) in deep neural networks, demonstrating that NC emerges from gradient flow dynamics under specific NTK conditions. The work identifies a training invariant whose state is shown to be a necessary condition for the full emergence of NC.

Key insights

  • Neural Collapse (NC) can be proven to emerge directly from the gradient flow dynamics of deep neural networks under the assumption of a block-structured, aligned empirical NTK.
  • A specific training invariant dictates whether NC occurs, with its value and alignment acting as a quantifiable necessary condition for the phenomenon.
  • Deep neural network training error reduction proceeds in distinct, interpretable phases, starting with global mean errors, then class mean errors, and finally individual sample errors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.