Reproduce “EditEmoTalk: Controllable Speech-Driven 3D Facial Animation with Continuous Expression Editing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Diqiong Jiang, Kai Zhu, Dan Song, Jian Chang, Chenglizhao Chen, Zhenyu Wu

A framework enables controllable speech-driven 3D facial animation with continuous expression editing by learning a boundary-aware semantic embedding for nuanced emotion control. This approach achieves superior geometric accuracy, lip synchronization, and emotional consistency across multiple datasets, with significantly lower Vertex Error and Lip Vertex Error compared to existing methods.

Key insights

  • Learning and leveraging the geometric structure of emotion space, specifically the normal directions to inter-emotion decision boundaries, enables the construction of a continuous emotion manifold for fine-grained control.
  • A dual-train strategy, which alternates between training with ground-truth and edited emotion embeddings, is crucial for simultaneously achieving accurate speech-driven animation and flexible, continuous emotion manipulation.
  • Explicitly modeling emotion consistency through a dedicated loss and projecting predicted expressions into an emotion embedding space ensures semantic alignment between intended emotion and generated facial motion dynamics.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.