Reproduce “VideoTetris: Towards Compositional Text-to-Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Jingmin Chen, Xintao Wang, Zhaochen Yu and 4 more

A framework from Peking University and Kuaishou Technology enables text-to-video generation that accurately composes multiple objects with specified attributes and spatial relationships, and produces coherent long videos with dynamic scene changes. The method achieves higher VBLIP-VQA and VUnidet scores compared to existing models in both short and progressive compositional long video generation tasks.

Key insights

  • Direct manipulation of cross-attention maps, rather than explicit layout conditions, provides an effective and training-free mechanism for achieving precise compositional control in diffusion models.
  • Improving the quality of training data through motion dynamics filtering and semantic prompt enrichment is crucial for the performance of auto-regressive long video generation models.
  • Aligning reference object features directly in the latent space of the pre-trained autoencoder, instead of relying on external encoders, significantly enhances inter-frame consistency and reduces artifacts in long video sequences.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.