Reproduce “Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

X.Y. Han, Vardan Papyan, David L. Donoho

An exact theory describes Neural Collapse (NC) dynamics in deep networks trained with Mean Squared Error (MSE) loss, empirically confirming its emergence and deriving closed-form predictions for its evolution. This framework, leveraging a novel MSE loss decomposition and "central path" model, quantifies the collapse of within-class variability and the convergence of class-means to a Simplex Equiangular Tight Frame (ETF).

Key insights

  • Neural Collapse (NC) is a robust phenomenon, empirically present under MSE loss similar to CE loss, sometimes exhibiting faster activation collapse and improved adversarial robustness.
  • The MSE loss can be effectively decomposed into terms directly related to NC properties, where the actual classifier's deviation from the least-squares optimal one becomes negligible early in training.
  • The dynamics of NC properties, such as within-class variability collapse (NC1) and class-mean convergence to a Simplex ETF (NC2), are precisely characterized by the growth and equalization of the singular values of the Signal-to-Noise Ratio (SNR) matrix.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.