Reproduce “JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anthony Chen, Naomi Ken Korem, Gal Zeevi, Tavi Halperin, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Or Patashnik and 1 more
Researchers from Tel Aviv University and Lightricks propose JUST-DUB-IT, a unified audio-visual diffusion model that adapts a foundational generative prior for end-to-end video dubbing. The model jointly translates audio and generates synchronized facial motion directly from a source video, achieving 100% success rates and superior temporal coherence (lower FVD) in complex scenarios compared to modular baselines.
Key insights
- Reframing video dubbing as a joint audio-visual generation task within a unified diffusion model allows for natural co-evolution of temporal structure, identity cues, and non-speech events, improving coherence.
- Leveraging the generative capabilities of an audio-visual foundation model to synthesize high-quality, identity-consistent multilingual training data can effectively mitigate data scarcity challenges for complex dubbing tasks.
- Lightweight adaptation mechanisms like LoRA are effective for steering powerful pre-trained multimodal foundation models to perform complex, task-specific video editing while maintaining robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.