Reproduce “DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Guo, Fulong Ye, Qichao Sun, Liyang Chen, Bingchuan Li, Pengze Zhang, Jiawei Liu, Songtao Zhao and 2 more
DreamID-Omni introduces a unified framework for human-centric audio-video generation, editing, and animation, integrating previously isolated tasks within a single model. This approach achieves precise and disentangled control over multiple character identities and voice timbres, demonstrating superior performance across generation, editing, and animation benchmarks, including significant reductions in speaker confusion to 0.08 in multi-person scenarios.
Key insights
- Human-centric audio-video generation, editing, and animation tasks share a fundamental objective of mapping static identity anchors onto a dynamic spatio-temporal canvas, making them suitable for a single, unified framework.
- Precise and disentangled control over multiple human subjects requires addressing entanglement at both signal (via distinct positional embeddings) and semantic (via structured captions) levels to prevent identity-timbre binding failures and speaker confusion.
- A progressive training curriculum, starting with weakly-constrained generative tasks to build a robust prior before fine-tuning on strongly-constrained tasks, is essential for stable convergence and optimal performance across a unified multi-task model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.