Reproduce “DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory” on OpenResearch

About this paper

Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, Nan Duan

DRAGNUWA, developed by a collaboration including the University of Science and Technology of China and Microsoft Research Asia, presents a diffusion-based model for fine-grained video generation by simultaneously integrating text, an initial image, and object/camera trajectories. The system achieves robust open-domain trajectory control for arbitrary objects and complex motions, addressing limitations of prior fragmented approaches.

Key insights

• Fine-grained video generation necessitates the concurrent integration of semantic (text), spatial (image), and temporal (trajectory) controls, as each modality provides unique and complementary information.
• Open-domain trajectory control can be effectively achieved by dynamically sampling and enhancing trajectories from video optical flow, combined with a two-stage adaptive training approach that leverages dense information for stability before adapting to sparser inputs.
• The model inherently learns to generate various camera movements (e.g., zoom, pan) by simply modifying dragging trajectories, demonstrating an implicit understanding of spatiotemporal transformations beyond explicit motion modeling.

Summary by alphaXiv.