Reproduce “CineDub: Scaling End-to-End Video Dubbing to Multi-Speaker Dialogues with Coherent Sound Effects” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yusheng Dai, Kangdi Wang, Baolong Gao, Yuxuan Jiang, Weiqiang Wang, Qiuhong Ke, Jianfei Cai
CineDub introduces an end-to-end, diffusion-based model for multi-speaker video dubbing and coherent joint speech and sound effect generation from uncropped video. The system achieves a 68.0% relative error reduction in speaker misattribution (cpWER) on a new multi-speaker benchmark and delivers high-quality, acoustically coherent outputs.
Key insights
- Holistic visual features from SynchFormer, despite not being explicitly trained for lip localization, provide sufficient fine-grained lip synchronization cues and exhibit emergent attention-switching for active speaker tracking in multi-speaker settings.
- Semantic-bundled transcriptions, which couple speaker identity with utterance content, are critical for implicitly grounding utterances to corresponding speakers, significantly improving speaker-utterance assignment in multi-speaker dialogues.
- A curriculum learning strategy that first trains on ambient audio generation (V2A) before integrating speech (V2SA) is crucial for mitigating performance degradation and optimization conflicts when jointly generating speech and sound effects.
- Decoupling cross-attention for heterogeneous textual conditions (speech transcription vs. audio prompts) with independent branches prevents attention dilution and enables high-fidelity, coherent joint generation by allowing specialized attention patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.