Reproduce “HumANDiff: Articulated Noise Diffusion for Motion-Consistent Human Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tao Hu, Varun Jampani
Stability AI researchers developed HumANDiff, a method that integrates human body priors into latent video diffusion models to generate motion-consistent human videos. The approach uses articulated noise sampling, joint appearance-motion learning, and geometric motion consistency, achieving state-of-the-art quantitative results with an FVD of 84.10 and FID of 7.94 on the UBC Fashion dataset.
Key insights
- Embedding human-specific structural priors directly into the noise sampling process of diffusion models significantly enhances motion control and temporal consistency.
- Jointly training a model to predict both appearance and physical motion from structured noise improves motion coherence and enables the synthesis of fine, motion-dependent details like clothing wrinkles.
- Enforcing motion consistency in an articulated 3D space, rather than just 2D image space, effectively mitigates temporal flickering and maintains physical coherence in human video generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.