Reproduce “MotionShot: Adaptive Motion Transfer across Arbitrary Objects for Text-to-Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yanchen Liu, Yanan Sun, Zhening Xing, Junyao Gao, Kai Chen, Wenjie Pei

MotionShot introduces a training-free framework that enables adaptive motion transfer across arbitrary objects for text-to-video generation. It achieves this by employing a novel two-level alignment strategy, combining high-level semantic matching and low-level morphological warping, allowing for faithful motion reproduction while maintaining the target object's appearance and structure, even with significant discrepancies between reference and target.

Key insights

  • A two-level motion alignment strategy, combining high-level semantic and low-level morphological alignment, is crucial for effectively transferring motion between objects with significant appearance and structural differences.
  • A training-free framework can adaptively transfer motion by leveraging powerful pre-trained diffusion models and their feature spaces (Stable Diffusion, DINOv2) for robust correspondence matching and guidance.
  • The fusion of low-level spatial features (from SD U-Net) and high-level semantic features (from DINOv2) combined with structure-aware keypoint sampling provides a robust method for establishing fine-grained correspondences between dissimilar objects.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.