Reproduce “Kaleido: Open-Sourced Multi-Subject Reference Video Generation Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenxing Zhang, Jiayan Teng, Zhuoyi Yang, Tiankun Cao, Cheng Wang, Xiaotao Gu, Jie Tang, Dan Guo and 1 more
KALEIDO, an open-sourced framework, generates subject-consistent videos from multiple reference images and text prompts by employing a novel data construction pipeline and a Reference Rotary Positional Encoding (R-RoPE) strategy. The model achieved the highest S2V Consistency (0.723) and S2V Decoupling (0.319) scores among evaluated models, indicating improved subject fidelity and background disentanglement, with human raters preferring its video quality, prompt alignment, and consistency.
Key insights
- The strategic inclusion of cross-paired data in the training pipeline acts as an effective regularizer, compelling the model to decouple subjects from their original contexts and significantly improving subject-background disentanglement.
- R-RoPE's method of shifting spatial positional encoding for reference image tokens provides a simple yet highly effective mechanism to inject multiple image conditions without causing temporal confusion or spatial overlap, thus enhancing conditioning without complex architectural modifications.
- A combination of robust data engineering, including detailed segmentation, inpainting, and diverse augmentation, with an efficient conditioning strategy allows open-source S2V models to achieve performance comparable to, or even surpassing, leading closed-source systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.