Reproduce “MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kyeongmin Yeo, Yunhong Min, Minhyuk Sung

MUNI, a multimodal unified latent diffusion framework developed at KAIST, enables coherent any-to-any generation by jointly training encoders, decoders, and a shared flow-based prior with a novel routed objective. It achieves superior multimodal coherence in unconditional co-generation, improving Audio-Image Similarity by +31.85, and enhances alignment in many-to-one cross-modal tasks on a large-scale image-text-audio benchmark.

Key insights

  • Jointly training all components (encoders, decoders, and a flow-based prior) within a single objective allows the shared latent space to be optimally shaped for both reconstruction and prior modeling in a multimodal setting.
  • Using non-mixture aggregation rules (PoE or Hellinger) for subset posteriors, combined with target-detached self-reconstruction, encourages the latent variable to encode only cross-modal predictive information, promoting minimality and predictive sufficiency.
  • Training the prior specifically on latents inferred from full observations or 'leave-one-out' subsets is crucial for ensuring the prior can generate fully coherent joint samples in unconditional co-generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.