Reproduce “Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, Tali Dekel
Researchers at the Weizmann Institute of Science and NVIDIA Research present a zero-shot text-driven method for transferring motion from an input video to a new object or scene described by a text prompt, even across vastly different categories. The approach leverages Spatial Marginal Mean (SMM) features from a pre-trained text-to-video diffusion model and optimizes their pairwise temporal differences, outperforming existing baselines in both motion preservation and text fidelity.
Key insights
- Empirical analysis shows that full space-time features encode fine-grained pixel-level information, while SMM features serve as effective global per-frame descriptors, capturing object pose and scene layout with flexibility for appearance and shape changes.
- Optimizing for pairwise SMM differences effectively decouples motion from absolute appearance and fine-grained structure, enabling significant cross-category object changes while preserving underlying motion patterns.
- A specialized filtered noise initialization strategy (combining low-frequency components of inverted noise with new random high-frequency noise) is crucial for stable convergence and robust preservation of the input video's coarse structural layout.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.