Reproduce “VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sicheng Xu, Guojun Chen, Jiaolong Yang, Yizhong Zhang, Yu Deng, Steve Lin, Baining Guo
VASA-3D from Microsoft Research Asia enables the generation of lifelike, free-viewpoint 3D head avatars from a single portrait image, driven by audio in real time. The system renders 512x512 output at up to 75 FPS and surpasses prior 3D talking head methods in visual quality and lip-sync accuracy by extending the expressive capabilities of VASA-1.
Key insights
- The rich motion latent space learned by 2D talking face models from vast video datasets can be effectively translated to drive highly expressive 3D avatars.
- Synthetically generated data from a powerful 2D model provides a scalable and viable training source for single-shot 3D avatar customization.
- A combination of parametric model-driven base deformation with residual dense Gaussian deformations is crucial for achieving both structural coherence and fine-grained expressive details in 3D avatars.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.