Reproduce “Video Diffusion Models are Training-free Motion Interpreter and Controller” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeqi Xiao, Yifan Zhou, Shuai Yang, Xingang Pan
This research from S-Lab, NTU, and Peking University developed a training-free method to interpret and control motion in video diffusion models by identifying and leveraging intrinsic "Motion Features" (MOFT). Their approach successfully generates natural and faithful video motion, including precise point-drag manipulation, and demonstrates superior motion naturalness in user studies.
Key insights
- Video diffusion models implicitly encode robust motion information within their intermediate latent features, which can be explicitly isolated by removing content-related components.
- Only a small subset of specific feature channels within the diffusion model's U-Net strongly correlates with motion, providing an interpretable and efficient way to extract motion-aware information.
- Effective video motion control can be achieved in a training-free and architecture-agnostic manner by directly guiding latent optimization using the extracted Motion Features (MOFT).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.