Reproduce “MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Akio Hayakawa, Masato Ishii, Takashi Shibuya, Yuki Mitsufuji
MMDISCO, a framework from Sony AI, introduces a multi-modal discriminator-guided cooperative diffusion approach for joint audio and video generation by leveraging pre-trained single-modal diffusion models. This method enhances cross-modal alignment and single-modal fidelity with minimal computational overhead, yielding improved objective metrics and strong subjective user preference for generated content.
Key insights
- Joint multimodal generation can be effectively guided in a model-agnostic way by approximating cross-modal conditional score terms from the gradient of an optimally trained discriminator.
- A denoising regularization strategy is crucial for stabilizing discriminator-based guidance, preventing degradation of single-modal fidelity, especially when integrating disparate base models.
- It is feasible to achieve high-quality, aligned joint audio-video generation by treating powerful single-modal diffusion models as fixed black boxes, offering significant computational and flexibility advantages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.