Reproduce “UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hebeizi Li, Zihao Liang, Benyuan Sun, Zihao Yin, Xiao Sha, Chenliang Wang, Yi Yang
UniTalking presents a unified, end-to-end diffusion framework that generates high-fidelity, lip-synchronized talking portraits from diverse inputs. The model demonstrates superior audio naturalness, perceptual quality, and precise lip-synchronization compared to existing open-source approaches, while also enabling personalized control over visual identity and acoustic style.
Key insights
- A symmetric dual-stream architecture within an MM-DiT, combined with a Joint Attention mechanism, effectively learns fine-grained intra- and inter-modal dependencies for precise audio-video alignment.
- Leveraging a strong pre-trained video foundation model while progressively training the audio branch from scratch is critical for stable convergence and high-quality multi-modal outputs.
- Augmented cross-attention for multi-modal conditions (image and reference audio) allows for personalized talking portrait generation that preserves visual identity and clones acoustic style within a unified framework.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.