Reproduce “Modelling the influence of data structure on learning in neural networks: the hidden manifold model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sebastian Goldt, Marc Mézard, Florent Krzakala, Lenka Zdeborová

The Hidden Manifold Model (HMM) is introduced to model high-dimensional data with lower-dimensional intrinsic structure, enabling an analytical description of two-layer neural network learning dynamics. This framework demonstrates how data structure influences phenomena like hidden unit specialization, learning curve plateaus, and the emergence of 'easy' and 'hard' examples in memorization.

Key insights

  • The Gaussian Equivalence Property (GEP) provides a powerful analytical tool, allowing for precise calculation of learning dynamics on structured data, even with deterministic feature matrices.
  • Learning in the HMM reveals a 'specialization phenomenon' where student hidden units become strongly correlated with individual teacher units during training, concurrent with decorrelation among student units.
  • The HMM effectively reproduces complex empirical learning phenomena, including the presence of plateaus in learning curves that signify increasing function complexity and the distinct memorization patterns of 'easy' and 'hard' examples.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.