Reproduce “Open-Sora Plan: Open-Source Large Video Generation Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye and 16 more
The Open-Sora Plan project introduces an open-source framework for generating high-resolution, long-duration videos from various user inputs, re-implementing capabilities akin to proprietary models. It integrates a Wavelet-Flow VAE for efficient latent compression and a Joint Image-Video Skiparse Denoiser, achieving strong performance in video quality and control.
Key insights
- Efficient latent compression, as demonstrated by WF-VAE with its Causal Cache, is crucial for handling long, high-resolution videos without quality loss during tiled inference.
- A carefully designed 3D attention mechanism with Skiparse Attention can effectively balance powerful spatiotemporal modeling with computational efficiency in large video diffusion transformers.
- A meticulous multi-dimensional data curation pipeline, including aesthetic and technical quality filtration, significantly impacts the performance of large video generation models.
- Auxiliary strategies like adaptive gradient clipping, LLM-powered prompt refinement, and optimized data sampling are vital for stable training and improved results in large-scale video generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.