Reproduce “A new initialisation to Control Gradients in Sinusoidal Neural network” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Andrea Combette, Antoine Venaille, Nelly Pustelnik

Researchers at ENSL, CNRS UMR 5672 Lyon developed an initialization strategy for sinusoidal neural networks (SIRENs) that theoretically controls gradient propagation and the network's frequency spectrum with increasing depth. This initialization, particularly the sigma_a=0 variant, yields lower generalization error, reduces spurious high-frequency artifacts in deep networks, and demonstrates improved performance across various tasks including function fitting, image reconstruction, and solving partial differential equations.

Key insights

  • Precise closed-form initialization parameters can be derived for deep sinusoidal networks by analyzing the statistical properties of pre-activations and gradients, enabling stable gradient propagation.
  • Controlling the pre-activation variance to approach zero ($\sigma_a=0$) prevents spectral broadening and aliasing in deep SIRENs, allowing for more accurate representation of high-frequency content without generating artifacts.
  • The proposed initialization influences early training dynamics through the Neural Tangent Kernel (NTK), leading to linear scaling of NTK eigenvalues with depth and stable input-space gradients, which correlates with improved generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.