Reproduce “One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ting-Chun Wang, Arun Mallya, Ming-Yu Liu
Researchers from NVIDIA Corporation developed a one-shot neural talking-head synthesis system that generates free-view videos with superior visual quality and enables up to a 10x bandwidth reduction for video conferencing compared to H.264. This is achieved through a novel unsupervised 3D keypoint representation that disentangles identity from motion.
Key insights
- Unsupervised learning can effectively discover 3D keypoints that disentangle a person's canonical geometry (identity) from transient head pose and facial expressions, crucial for both novel view synthesis and compact representation.
- Warping 3D appearance features based on decomposed 3D keypoints and a learned motion field enables the generation of realistic talking-head videos with controllable head poses, overcoming limitations of prior 2D methods.
- Representing facial motion through a small set of 3D keypoints, head pose, and expression deformations provides a highly compact encoding that facilitates significant bandwidth reduction while maintaining high perceptual quality for real-time applications like video conferencing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.