Reproduce “IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bo Chen, Tao Liu, Qi Chen, Xie Chen, Zilong Zheng
IMTalker presents an efficient framework for audio-driven talking face generation, leveraging an implicit motion transfer mechanism within a unified latent space and an Identity-Adaptive Module to preserve identity during complex facial animation. This approach achieves state-of-the-art results across video-driven and audio-driven benchmarks, operating at real-time speeds of 40-42 FPS for 512x512 resolution output.
Key insights
- Implicit motion transfer through cross-attention effectively handles complex and non-rigid facial dynamics more robustly than local explicit warping.
- Personalizing motion representations with an identity-adaptive module is crucial for maintaining identity fidelity, especially in cross-identity reenactment scenarios.
- Flow-matching models offer an efficient and controllable method for generating precise audio-to-motion representations, balancing fidelity and computational cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.