Reproduce “BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengyuan Shi, Jiaxi Gu, Hang Xu, Songcen Xu, Wei Zhang, Limin Wang
BIVDiff presents a training-free framework that leverages existing image diffusion models (IDMs) and video diffusion models (VDMs) for general-purpose video synthesis. The method generates diverse video content, including controllable generation, editing, and inpainting, by ensuring strong temporal consistency via a mixed inversion technique that bridges the two model types.
Key insights
- A sequential bridging strategy, separating content generation (IDM) from temporal coherence enforcement (VDM), proves more effective than direct fusion or alternating denoising.
- Mixed Inversion is crucial for mitigating distribution shifts between different diffusion models, enabling stable VDM processing while retaining IDM-generated details.
- The framework efficiently reuses powerful pre-trained IDMs and VDMs, avoiding costly training or fine-tuning, thereby democratizing advanced video synthesis capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.