Reproduce “Inherent Weight Normalization in Stochastic Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Georgios Detorakis, Sourav Dutta, Abhishek Khanna, Matthew Jerry, Suman Datta, Emre Neftci
NSM uses multiplicative noise and binary threshold neurons so firing probabilities depend on L2-normalized incoming weights without batch statistics. Its stochastic forward pass supports learning and inference, while gradients pass through activation probabilities. On MNIST, Bernoulli-noise NSM reports 0.775% test error versus 0.880% for a matched deterministic CNN. However, the results are GPU simulations, and NSM is not uniformly better.
Key insights
- Always-on multiplicative stochasticity in binary threshold neurons naturally induces a self-normalizing effect that mirrors the reparameterization used in Weight Normalization.
- This inherent normalization stabilizes pre-activation distributions across network layers during training, effectively mitigating internal covariate shift without requiring explicit batch-dependent operations.
- Simple hard-firing threshold neurons, when combined with controlled multiplicative noise, are sufficient for constructing effective deep neural networks and are highly compatible with the inherent stochasticity of emerging neuromorphic hardware.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.