Reproduce “Wide stochastic networks: Gaussian limit and PAC-Bayesian training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eugenio Clerico, George Deligiannidis, Arnaud Doucet
The authors develop a PAC-Bayesian training method that optimizes expected 0–1 error using a Gaussian approximation to the outputs of a stochastic, single-hidden-layer network. This makes the expected error differentiable with respect to the output distribution, avoiding a surrogate loss. On full MNIST, it achieved a 0.2807 final bound and 0.1083 test error, outperforming tested surrogate methods in this setup.
Key insights
- Overparameterized stochastic neural networks with sufficient width exhibit an output distribution that can be accurately approximated by a multivariate Gaussian.
- The "lazy training" phenomenon, often observed in wide networks, is naturally encouraged by PAC-Bayesian optimization, preserving the network's Gaussian output behavior.
- Approximating the network's output as Gaussian allows for explicit and differentiable computation of the expected 01-loss, enabling more direct and principled optimization of PAC-Bayesian bounds.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.