Reproduce “FactorPortrait: Controllable Portrait Animation via Disentangled Expression, Pose, and Viewpoint” on OpenResearch

About this paper

Jiapeng Tang, Kai Li, Chengxiang Yin, Liuhao Ge, Fei Jiang, Jiu Xu, Matthias Nießner, Christian Häne and 3 more

FactorPortrait, developed by Meta Reality Labs and the Technical University of Munich, presents a video diffusion method for animating portraits from a single image. It achieves high-fidelity synthesis with disentangled control over facial expressions, head pose, and continuous camera viewpoint, outperforming prior methods with superior image quality and identity preservation across various animation tasks.

Key insights

• Implicitly defined 128-D latent expression codes are crucial for capturing fine-grained, nuanced facial dynamics, including wrinkles and mouth interior movements, outperforming 2D facial landmarks.
• Normal maps provide a robust, dense, and pixel-aligned representation for head pose conditioning, leading to more accurate pose control and superior image quality.
• A hybrid data curation strategy, particularly augmenting real data with synthetic datasets like DynamicSweep and ViewSweep, is essential for training models capable of complex, simultaneous control over camera viewpoint, pose, and expression.

Summary by alphaXiv.