Reproduce “Video Motion Transfer with Diffusion Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati

DiTFlow introduces an optimization-based, training-free method for video motion transfer in Diffusion Transformers, leveraging patch-wise Attention Motion Flow extracted from DiT attention mechanisms. This approach achieved higher motion fidelity and prompt adherence in user studies compared to existing methods, and demonstrates zero-shot motion transfer capabilities by optimizing positional embeddings.

Key insights

  • Diffusion Transformers' unified spatio-temporal attention mechanisms contain rich, integrated motion signals that can be effectively extracted as patch-wise Attention Motion Flow (AMF).
  • Optimizing internal DiT components (latents or positional embeddings) against a reference AMF enables precise, fine-grained motion transfer without requiring model retraining.
  • Optimizing positional embeddings within DiTs can achieve a novel zero-shot motion transfer, better disentangling motion from content and preserving content fidelity when applying motion to new prompts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.