Reproduce “EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, Xun Cao
Researchers from Nanjing University and SenseTime Research introduced EAMM, a system that generates emotionally expressive talking faces from a single image by leveraging an emotion-aware motion model. The model extracts dynamic emotion patterns from a separate video, enhancing realism and achieving higher emotion classification accuracy and naturalness in synthesized facial animations.
Key insights
- Dynamic facial emotion can be effectively modeled and transferred as an additive motion displacement, separate from speech-driven movements.
- Unsupervised key-point and Jacobian representations provide a flexible means to capture and manipulate local facial dynamics without needing explicit 3D models.
- Strategic data augmentation, including mouth occlusion and temporal/geometric perturbations, is crucial for disentangling emotional cues from speech, identity, and pose variations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.