Reproduce “MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Richard, Michael Zollhoefer, Yandong Wen, Fernando de la Torre, Yaser Sheikh
MeshTalk animates detailed 3D faces from speech, guiding mouth motion with audio while allowing plausible upper-face expression. It learns motion codes from audio and recorded expressions, then generates animations with an audio-conditioned autoregressive model. In full-face comparisons with VOCA, MeshTalk was rated better or equal in 75.3% of trials, including ties; this does not establish real-time capability.
Key insights
- A categorical latent space is more effective than a continuous one for representing and sampling diverse, sparse facial expressions, particularly for upper-face movements, preventing them from collapsing to mean expressions.
- Explicitly disentangling audio-correlated and audio-uncorrelated facial information via a cross-modality loss is critical for simultaneous accurate lip synchronization and plausible, independent upper-face animation.
- An autoregressive model conditioned on audio can effectively synthesize varied and natural facial expressions, especially for regions not directly driven by speech, enabling generalized animation for arbitrary identities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.