Reproduce “MOVi: Training-free Text-conditioned Multi-Object Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aimon Rahman, Jiang Liu, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Yusheng Su, Vishal M. Patel and 2 more
MOVi, a training-free framework developed by researchers at Johns Hopkins University and Advanced Micro Devices, enhances multi-object video generation in existing text-to-video diffusion models. It leverages large language models for trajectory planning and manipulates diffusion noise reinitialization and cross-attention to produce distinct objects with dynamic, coherent motions, achieving a 100% Dynamic Degree and an FVD of 295.3, outperforming several state-of-the-art baselines.
Key insights
- Low-frequency noise components in diffusion models significantly contribute to an object's layout and shape, making them suitable for trajectory-guided manipulation to control object movement and placement.
- Large Language Models (LLMs) possess sufficient open-world knowledge to effectively plan dynamic object trajectories from text-only prompts, acting as an implicit conditioning mechanism for motion control.
- Combining targeted manipulation of both initial noise (for motion coherence) and cross-attention (for semantic distinctness) within a training-free framework can overcome the challenges of multi-object generation in T2V models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.