Reproduce “DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong and 1 more
DynamiCrafter provides a framework for animating open-domain still images into dynamic videos, leveraging pre-trained text-to-video diffusion models via a dual-stream image injection paradigm. The method generates temporally coherent videos that preserve input image fidelity while enabling text-based motion control, outperforming current open-source I2V models on quantitative metrics and exhibiting strong qualitative performance against commercial tools.
Key insights
- Effective open-domain image animation requires both high-level semantic understanding for plausible dynamics and low-level visual detail guidance for fidelity, achieved through a dual-stream image injection paradigm.
- A multi-stage training strategy is crucial for successfully adapting pre-trained text-to-video (T2V) diffusion models for image animation without disrupting their essential temporal priors.
- Text prompts can enable fine-grained motion control for image animation, allowing users to guide the specific dynamics generated in an open-domain setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.