Reproduce “UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos” on OpenResearch

About this paper

Yuxuan Zhang, Haozhong Xiong, Jiayi Song, Jinpeng Yu, Yang Shi, Jiaming Liu, Ruihua Huang, Liwei Wang

UniSwap presents a training-free framework for streaming joint audio-visual identity replacement in talking videos, transferring reference appearance and vocal timbre while preserving source motion, background, and linguistic content. The system generates content at 13.6 frames per second on a single NVIDIA H100 GPU, maintaining stable identity and synchronization over long durations and outperforming cascaded baselines.

Key insights

• Joint audio-visual identity replacement within a single diffusion transformer can achieve superior synchronization and consistency compared to cascaded unimodal approaches.
• Synthetic training data, generated by creating identity-altered versions of original videos and using them as source-target pairs, effectively addresses the challenge of data scarcity for complex multi-modal generative tasks.
• Adapting absolute positional encodings and caching mechanisms (Feature-RoPE Decomposition) is crucial for maintaining identity and quality over extended, real-time autoregressive generation.
• Distilling a multi-step diffusion model into an efficient student model via self-forcing DMD significantly reduces inference costs and mitigates exposure bias for streaming applications.

Summary by alphaXiv.