Reproduce “Random Weight Factorization Improves the Training of Continuous Neural Representations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sifan Wang, Hanwen Wang, Jacob H. Seidman, Paris Perdikaris

A new technique, Random Weight Factorization (RWF), re-parameterizes neural network weights to improve the training of continuous neural representations by modifying the loss landscape and introducing self-adaptive learning rates. This approach consistently yields superior accuracy and faster convergence across diverse applications like image regression, 3D shape representation, and scientific machine learning tasks, with marginal computational overhead.

Key insights

  • RWF fundamentally alters the geometry of the loss landscape, allowing the effective distance between parameter configurations to become arbitrarily small for sufficiently large scale factors, facilitating easier optimization.
  • The factorization implicitly assigns a self-adaptive learning rate to each individual neuron, dynamically adjusting the update magnitude during training to better capture both low and high-frequency information.
  • RWF generalizes existing techniques like weight normalization and SIREN's fixed weight scaling by introducing a trainable, randomly initialized scale factor for each neuron.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.