Reproduce “OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models” on OpenResearch

About this paper

Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, Chao Liang

ByteDance researchers developed OmniHuman-1, a Diffusion Transformer-based framework that scales human animation models by training with mixed, multi-modal conditions. This unified model generates highly realistic and flexible human videos from a single image and various driving signals, outperforming specialized methods in realism, lip-sync, and gesture richness.

Key insights

• Scaling human animation data is feasible by effectively mixing motion-related conditions, allowing models to learn from a wider range of 'noisy' data that would typically be discarded.
• Introducing stronger conditioning signals (e.g., pose) later in the training process and assigning them lower training ratios (e.g., audio 50%, pose 25%) prevents them from dominating weaker signals and improves overall motion learning.
• A single, unified model can effectively handle diverse human animation scenarios—from face close-ups to full-body, and talking to singing with object interaction—across various driving modalities, surpassing the limitations of specialized models.

Summary by alphaXiv.