Reproduce “Scaling Zero-Shot Reference-to-Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zijian Zhou, Shikun Liu, Haozhe Liu, Haonan Qiu, Zhaochong An, Weiming Ren, Zhiheng Liu, Xiaoke Huang and 9 more
Meta AI developed Saber, a framework for zero-shot Reference-to-Video generation that leverages a masked training strategy on general video-text datasets, eliminating the need for specialized R2V data. It achieves superior identity consistency and overall performance on benchmarks like OpenS2V-Eval compared to methods trained on costly R2V datasets.
Key insights
- High-quality R2V generation, including robust identity preservation, is achievable in a zero-shot setting by training solely on abundant video-text pair datasets.
- Simulating reference conditions through a diverse masked training strategy, where masks are generated from the training video itself, effectively teaches identity consistency without specific R2V triplets.
- Mask augmentation, combined with a specialized attention mechanism, is crucial for preventing "copy-paste" artifacts and naturally integrating reference subjects into generated videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.