Reproduce “PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chaonan Ji, Jinwei Qi, Sheng Xu, Peng Zhang, Bang Zhang

PortraitDirector, a hierarchical disentanglement framework from Tongyi Lab, facilitates controllable and real-time facial reenactment. It allows independent manipulation of head pose, eye, mouth, and emotion components, achieving 512x512 resolution processing at 20 frames per second with 800ms latency.

Key insights

  • Modeling facial motion as a multi-layered, compositional signal rather than a monolithic, entangled one enables more robust and granular control over individual attributes.
  • Explicit, architectural separation of facial components at the physical input level (e.g., cropped regions for eyes/mouth, dedicated pose adapter) is more effective for disentanglement than post-hoc decomposition of a fused latent space.
  • Global emotion can be effectively isolated by treating it as a low-frequency, temporally smooth component and using an information bottleneck mechanism to filter it from local movements.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.