Reproduce “UniVG: Towards UNIfied-modal Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ludan Ruan, Lei Tian, Chuanwei Huang, Xu Zhang, Xinyan Xiao
Baidu Inc. developed UniVG, a unified system capable of generating videos from flexible combinations of text and image inputs. It achieves the lowest Fréchet Video Distance on the MSR-VTT benchmark by categorizing tasks by "generative freedom" and introducing Biased Gaussian Noise for content preservation during generation.
Key insights
- Categorizing video generation tasks by "generative freedom" (high-freedom vs. low-freedom) provides a novel conceptual framework for designing more adaptable models.
- The Multi-condition Cross Attention (MCA) mechanism allows a single base model to flexibly interpret and combine diverse input modalities, such as text and images.
- Biased Gaussian Noise (BGN) effectively addresses the challenge of content preservation and detail generation in conditional diffusion models for low-freedom tasks by guiding the noise towards the conditional input.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.