Reproduce “Dense Sample Deep Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Stephen Josè Hanson, Vivek Yadav, Catherine Hanson

Researchers at Rutgers University investigated the internal learning mechanisms of deep neural networks using a dense sample category task, revealing that learning proceeds through multiple logistic processes and proposing an auto-catalytic theory for feature emergence. The model achieved approximately 97% accuracy in classifying five facial identities, demonstrating complete category separation by 51 epochs and providing insights into feature construction dynamics.

Key insights

  • Deep Learning exhibits multi-factorial learning dynamics, where the overall learning curve is a composite of multiple, distinct hyperbolic processes (Logistic Learning Decomposition), with faster processes extracting separable features and slower ones refining the decision surface.
  • Category formation within deep networks involves an initial 'crystallization' phase where feature detectors for a specific category emerge and become sufficiently tuned, followed by a more gradual accumulation of variance.
  • Feature creation in DL networks may be analogous to auto-catalytic chemical reactions, leading to an initial slow induction period followed by an explosive, self-sustaining growth of structure and complex, high-fidelity representations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.