Reproduce “SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Di Qiu, Zhengcong Fei, Rui Wang, Jialin Bai, Changqian Yu, Mingyuan Fan, Guibin Chen, Xiang Wen
Skywork AI developed SkyReels-A1, a video diffusion transformer framework that generates high-fidelity and expressive portrait animations. It achieves strong identity preservation and realistic facial dynamics across diverse body proportions through expression-aware conditioning and a multi-stage training paradigm, outperforming open-source methods and competing with commercial products.
Key insights
- Utilizing high-precision 3D facial landmarks, extracted via a neural rendering approach, significantly enhances the accuracy and realism of transferring nuanced facial expressions.
- Projecting facial identity features into the text feature space of a pre-trained DiT effectively maintains consistent identity throughout animation with minimal parameter overhead.
- A multi-stage training strategy, gradually refining motion, identity, and then jointly optimizing all components, is crucial for achieving stable, high-fidelity, and generalized portrait animation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.