Reproduce “Random Walk Initialization for Training Very Deep Feedforward Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

David Sussillo, L.F. Abbott

The authors introduce Random Walk Initialization to balance gradient changes in deep feedforward networks. At initialization, it scales layer-specific random weight matrices so the average change in log gradient norm is near zero, although individual gradients still fluctuate. In a 500-layer linear simulation, the mean log-norm stayed near zero. Task experiments show trainability, not that greater depth improves performance or generalization.

Key insights

  • The back-propagated gradient norm in feedforward networks undergoes a random walk, rather than the exponential decay or growth typically observed in recurrent networks.
  • The variance of this gradient log-norm random walk scales linearly with network depth and inversely with network width, implying wider networks are more stable.
  • Careful, theoretically-derived weight initialization, along with a layer-specific learning rate schedule, is sufficient to enable training of extremely deep FFNs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.