Reproduce “Splat-Portrait: Generalizing Talking Heads with Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tong Shi, Melonie de Almeida, Daniela Ivanova, Nicolas Pugeault, Paul Henderson
Splat-Portrait, developed at the University of Glasgow, introduces a 3D Gaussian Splatting-based approach to synthesize realistic and generalizable 3D talking head videos from a single image and audio. The method achieves higher fidelity in "same-identity" generation (PSNR 23.87, SSIM 0.814, FID 25.58) and superior identity preservation in "cross-identity" settings (CSIM 0.726, FID 28.62) compared to existing methods, without relying on 3D supervision or complex motion priors.
Key insights
- The explicit representation of 3D Gaussian Splatting allows for effective disentanglement of static geometry and dynamic motion, leading to more natural and controllable talking head animations.
- A two-stage training approach (static reconstruction followed by dynamic audio-driven motion) is crucial for learning robust and generalizable 3D talking heads from monocular video without 3D supervision.
- Score distillation from powerful 2D diffusion models can significantly improve the quality and realism of rarely seen extreme viewpoints during training.
- Explicitly modeling and inpainting the background alongside 3D splats creates seamless scene reconstructions and eliminates common "floating head" artifacts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.