Reproduce “Variational Nested Dropout” on OpenResearch

About this paper

Yufei Cui, Yu Mao, Ziquan Liu, Qiao Li, Antoni B. Chan, Xue Liu, Tei-Wei Kuo, Chun Jason Xue

Researchers developed Variational Nested Dropout (VND), a probabilistic operation that learns the ordered importance of neural network components and latent dimensions from data. This enables the creation of adaptive Bayesian Nested Neural Networks (BN3) with improved uncertainty quantification and deterministic distilled versions, alongside enhancing structured latent representations in generative models for better diversity and hierarchical generation.

Key insights

• Learning the importance and ordering of network components and latent dimensions dynamically through a variational approach yields more adaptable, accurate, and robust models compared to fixed, heuristic-based methods.
• The novel Downhill distribution provides an effective and differentiable mechanism to model ordered discrete choices, enabling end-to-end learning for both nested network architectures and structured latent spaces.
• Integrating a full Bayesian treatment into nested networks not only allows for data-driven adaptive compression but also provides principled, well-calibrated uncertainty estimates for enhanced model trustworthiness.

Summary by alphaXiv.