Reproduce “VideoComposer: Compositional Video Synthesis with Motion Controllability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao and 1 more
VideoComposer extends compositional generation from images to videos, enabling fine-grained control over spatial and temporal elements through diverse conditions like text, sketches, and direct motion vectors. The model generates coherent videos with enhanced temporal consistency and precise motion control across various synthesis tasks.
Key insights
- The compositional generation paradigm, successful in image synthesis, can be effectively adapted to video by thoughtfully decomposing video content into independently controllable spatio-temporal conditions.
- Explicit motion vectors from compressed video provide a direct and efficient mechanism for precise temporal control over object movements and dynamic patterns.
- A unified Spatio-Temporal Condition Encoder is crucial for integrating diverse and sequential input conditions, ensuring temporal consistency and structural coherence in the synthesized video outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.