Reproduce “VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yexin Liu, Yunlong Lin, Zehan Wang, Guangxin He, Xinli Xu, Jiajun Zha, Yaofu Liu, Yueze Wang and 6 more
VidDub edits video dubbing by guiding mouth motion with replacement audio while preserving the rest of the source. For paired audio, it retains full-frame reconstruction; for replacement audio, it masks mouth-region reconstruction losses and uses audio–visual synchronization supervision. A two-step full-context student is distilled from a teacher. Source-mouth masking improves cross-audio Sync-C by 0.833; it targets offline, not streaming, dubbing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.