Reproduce “CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhao Wang, Aoxue Li, Lingting Zhu, Yong Guo, Qi Dou, Zhenguo Li
CustomVideo provides a new framework for customized text-to-video generation, effectively creating identity-preserving videos featuring multiple subjects. It ensures consistent co-occurrence and disentanglement of subjects, achieving state-of-the-art performance on the CustomStudio dataset.
Key insights
- Fine-tuning a T2V diffusion model solely with concatenated multi-subject images is effective for enforcing subject co-occurrence, outperforming mixed training strategies.
- Explicit attention control, with both positive alignment and negative suppression guided by object masks, is vital for disentangling visually similar subjects and preserving individual identities in latent space.
- Parameter-efficient fine-tuning of cross-attention layers, while keeping the temporal transformer fixed, allows for effective identity learning without compromising the base model's motion generation capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.