Reproduce “Spatially heterogeneous learning by a deep student machine” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hajime Yoshino

Hajime Yoshino developed an exactly solvable statistical mechanics model for deep neural networks in a "dense limit" to explain how over-parametrized systems achieve generalization. This work demonstrates that learning is spatially heterogeneous, with "crystalline" regions at network boundaries enclosing a "liquid" core, and finds that generalization error converges to a depth-independent value in sufficiently deep networks.

Key insights

  • Learning within deep networks is spatially heterogeneous, characterized by "crystalline" regions (successful teacher inference) emerging from input/output layers and a "liquid" (de-correlated) region in the central layers that persists with over-parametrization.
  • Generalization error in sufficiently deep networks becomes independent of depth (L), converging to a finite value even as the central layers remain largely decorrelated from the teacher.
  • Finite network width (N) and reduced effective data dimension (D) significantly improve generalization by enhancing internal network correlations through loop corrections to the dense limit.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.