Reproduce “BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhaoyang Li, Dongjun Qian, Kai Su, Qishuai Diao, Xiangyang Xia, Chang Liu, Wenfei Yang, Tianzhu Zhang and 1 more

BindWeave generates subject-consistent videos by integrating a Multimodal Large Language Model for deep cross-modal reasoning to guide a Diffusion Transformer, surpassing previous shallow fusion methods. The framework achieved the highest overall score of 57.61% and a NexusScore of 46.84% on the OpenS2V-Eval benchmark, demonstrating improved identity fidelity and temporal coherence.

Key insights

  • Deep cross-modal reasoning, facilitated by an MLLM, can effectively overcome the limitations of shallow multimodal fusion in understanding complex interactions between textual prompts and visual subjects.
  • An MLLM can generate subject-aware hidden states that disentangle roles, attributes, and spatio-temporal relations, providing robust high-level guidance for video synthesis.
  • A multi-level conditioning strategy, combining MLLM-derived reasoning with semantic (CLIP) and low-level appearance (VAE) features, is essential for ensuring both high fidelity and strong subject consistency in generated videos.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.