Reproduce “EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier Perrotin
EMODY Flow extends the frozen Qwen-3 Omni model with flow-matching generators that produce speech-synchronized SMPL-X body motion and FLAME facial animation from the model’s internal Mimi audio representations. An auxiliary emotion classifier makes discrete emotion controls effective: body-motion outputs become more separable by emotion, while the base model achieves FGD 0.302, beat correlation 0.853, and diversity 24.62 on BEAT2.
Key insights
- A high-dimensional speech representation can dominate a low-dimensional emotion label, causing a generative model to produce nearly identical motion across emotion settings.
- Training through a motion-based auxiliary classifier encourages emotion-dependent body movement without requiring explicit latent disentanglement, although it reduces some general gesture-quality metrics.
- Emotion control is more effective for body motion than facial animation, where lip synchronization and upper-face expressiveness compete for model capacity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.