Reproduce “Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders” on OpenResearch

About this paper

Rogelio A Mancisidor, Robert Jenssen, Shujian Yu, Michael Kampffmeyer

CoDE-VAE introduces a multimodal variational autoencoder framework that aggregates information from diverse data modalities by explicitly modeling the stochastic dependence between expert distributions. The method achieves a superior balance between generative coherence and quality, provides more precise log-likelihood estimations, and significantly minimizes the generative quality gap, maintaining unimodal VAE quality even with multiple modalities.

Key insights

• Explicitly modeling the stochastic dependence between modality-specific expert distributions, through their correlated estimation errors, is critical for achieving more accurate and robust multimodal information aggregation.
• Dynamically learning the contribution of different modality subsets to the ELBO improves optimization efficiency by adaptively prioritizing more informative or reliable combinations of modalities.
• The CoDE method generalizes existing approaches like Product of Experts, subsuming it as a special case when the correlation between expert errors is zero, offering a more flexible and theoretically sound aggregation strategy.

Summary by alphaXiv.