Reproduce “Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinbo Xing, Zeyinzi Jiang, Yuxiang Tuo, Chaojie Mao, Xiaotang Gai, Xi Chen, Jingfeng Zhang, Yulin Pan and 10 more
The Wan-Weaver model generates interleaved text and images by decoupling the task into a Planner (VLM) for narrative coherence and dense prompt generation, and a Visualizer (Diffusion Transformer) for image synthesis. This architecture, combined with a decoupled training strategy leveraging synthetic and reference-guided data, achieves strong performance on multi-turn multi-modal generation benchmarks.
Key insights
- Decomposing interleaved generation into textual planning and visual consistency modeling enables robust performance without relying on extensive real interleaved training data.
- Utilizing large-scale synthetic "textual-proxy interleaved data" effectively trains the Planner to manage modality transitions and generate detailed visual guidance (dense prompts).
- Strategic use of diverse visualizer training data, including multi-image reference data derived from videos and homologous images, enhances long-range visual coherence for consistent object identity and style.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.