Reproduce “Any-to-Any Generation via Composable Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, Mohit Bansal
CoDi introduces the first 'any-to-any' multimodal generative AI model, developed by the University of North Carolina at Chapel Hill and Microsoft Azure Cognitive Services Research, capable of simultaneously generating coherent combinations of text, image, video, and audio from arbitrary multimodal inputs. This model unifies generation across previously separate modalities, outperforming independent generation in terms of cross-modal coherence.
Key insights
- Text can serve as an effective universal bridge for aligning diverse modalities in a shared latent space, mitigating the combinatorial explosion of direct pairwise multimodal training.
- Coordinating diffusion processes through cross-modal attention and shared latent spaces enables simultaneous and coherent multimodal generation.
- A unified 'any-to-any' generative model is achievable by combining specialized modality-specific diffusers with a flexible cross-modal alignment framework.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.