Reproduce “BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weixi Feng, Chao Liu, Sifei Liu, William Yang Wang, Arash Vahdat, Weili Nie

BlobGEN-Vid introduces a novel framework that enhances compositional and fine-grained control in text-to-video generation through the use of blob video representations. The system achieved a 20% improvement in spatial controllability (0.6119 mIOU) on YoutubeVIS-2021 compared to prior methods and outperformed leading proprietary models in compositional accuracy benchmarks.

Key insights

  • Decomposing video into object-level 'blob video representations' provides a more robust and versatile primitive than bounding boxes for detailed spatial, temporal, and semantic control.
  • A specialized Masked 3D Self-Attention module is critical for enforcing object-level temporal consistency by allowing objects to attend to themselves across frames.
  • Context interpolation of sparse blob descriptions enables smooth semantic transitions and efficient annotation while maintaining high prompt alignment and video quality.
  • Integrating LLMs for automated blob planning significantly reduces manual effort, making advanced compositional control more accessible for complex video generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.