Reproduce “Deep Variational Sufficient Dimensionality Reduction” on OpenResearch

About this paper

Ershad Banijamali, Amir-Hossein Karimi, Ali Ghodsi

The Deep Variational Sufficient Dimensionality Reduction (DVSDR) model, based on variational autoencoders, learns low-dimensional data representations that preserve all information for a label variable. This framework achieved a 0.80% classification error rate on the fully labeled MNIST dataset and demonstrated effective generation of new samples for both MNIST and Fashion MNIST.

Key insights

• Recasting the SDR problem within a deep variational learning framework allows for the discovery of complex, non-linear sufficient representations, leveraging the expressive power of deep neural networks.
• A clear graphical model interpretation of SDR, where the latent variable z generates both the observation x and the label y, provides a theoretical foundation for the deep learning approach.
• By incorporating a classification loss term directly into the VAE's Evidence Lower Bound (ELBO), the model explicitly enforces the SDR criterion while simultaneously acquiring generative capabilities.
• The DVSDR framework can naturally extend to semi-supervised settings, enabling the model to learn more robust latent spaces by leveraging abundant unlabeled data alongside limited labeled data.

Summary by alphaXiv.