Reproduce “Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Teng Hu, Zhentao Yu, Guozhen Zhang, Zihan Su, Zhengguang Zhou, Youliang Zhang, Yuan Zhou, Qinglin Lu and 1 more
Harmony, a generative AI framework, introduces a new state-of-the-art in harmonized audio-video synthesis by systematically addressing challenges in joint diffusion processes. It achieves superior audio-visual synchronization, producing high-fidelity human speech with dynamically synchronized lip movements and realistic environmental sounds precisely matched to visual events.
Key insights
- Learning robust audio-visual correspondence benefits from complementary uni-directional generation tasks, which provide stable supervision and mitigate "Correspondence Drift" in joint diffusion.
- Effective audio-visual synchronization necessitates an architectural separation between mechanisms for fine-grained temporal alignment and those for maintaining holistic global style consistency.
- Classifier-Free Guidance can be extended to enhance cross-modal alignment by creating targeted
- negative anchors
- that specifically isolate and amplify synchronization signals during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.