Reproduce “Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chanhyuk Choi, Taesoo Kim, Donggyu Lee, Siyeol Jung, Taehwan Kim
Researchers developed Cross-Modal Emotion Transfer (C-MET) to generate expressive facial emotions in talking face videos by leveraging emotion semantic vectors derived from speech. The approach achieved state-of-the-art emotion classification accuracy, outperforming baselines by 14% on the MEAD dataset, and successfully synthesized a wider range of nuanced and "extended" emotions.
Key insights
- Emotion can be represented as semantic vectors, capturing continuous differences between emotional states for fine-grained cross-modal transfer.
- Combining contrastive learning with modality-specific encoders enables effective alignment of audio and visual emotion representations, bridging the domain gap.
- A Transformer encoder can accurately predict visual emotion semantic vectors from audio, facilitating disentangled emotion transfer for expressive talking faces.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.