Reproduce “ESGaussianFace: Emotional and Stylized Audio-Driven Facial Animation via 3D Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chuhang Ma, Shuai Tan, Ye Pan, Jiaolong Yang, Xin Tong
ESGaussianFace presents a 3D Gaussian Splatting framework for generating real-time, audio-driven facial animations that incorporate both controllable emotional expressions and arbitrary artistic styles. The system achieves an inference speed of 69.624 FPS, while demonstrating enhanced fidelity, lip synchronization, and emotional accuracy compared to prior methods.
Key insights
- An emotion-audio-guided spatial attention module (ESAM) is crucial for dynamically focusing on distinct facial regions for accurate audio-driven lip synchronization and emotion expression.
- A multi-stage training strategy, progressing from neutral lip movements to emotional and then stylized deformations, effectively handles the complexity of learning entangled features.
- Representing the face with explicit 3D Gaussians and a triplane encoding enables high-fidelity, 3D-consistent animation with real-time rendering capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.