Reproduce “Variational Bayesian Neural Networks via Resolution of Singularities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Susan Wei, Edmund Lau
Researchers from the University of Melbourne applied Singular Learning Theory (SLT) to Variational Inference (VI) for Bayesian Neural Networks (BNNs), developing a normalizing flow with a generalized gamma base distribution. Their empirical work demonstrates that this SLT-informed approach can yield lower variational free energy in specific BNN configurations and elucidates the observed disconnect between optimizing the variational objective and achieving improved predictive generalization.
Key insights
- The observed discrepancy between minimizing Variational Free Energy (VFE) and improving Variational Generalization Error (VGE) in BNNs is theoretically explained by the inequality of their asymptotic coefficients, λ_vfe ≠ λ_vge.
- Singular Learning Theory (SLT) provides the correct asymptotic form for posterior distributions in singular models, offering a principled framework for designing more appropriate variational families.
- An idealized SLT-informed variational family, constructed from generalized gamma distributions, exhibits a variational approximation gap that is bounded independently of sample size 'n' to leading order under certain conditions (multiplicity m=1).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.